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How to Read 
• This Report 



This report is designed primarily to be a series of 
"How To" booklets. 

1. How to operationalize a Learning Assessment 
Plan for a cooperative of school districts 
(Chapter 2); 

2. How to and what to acquire as hardware and 
software (Chapter 3); 

3. How to scan, score, and generate reports 
(Chapter 4); 

4. How to interpret and utilize the assessment 
results (Chapter 5); and 

5. How to do all of the above in an integrated 
manner with maximum efficiency, minimum 
headaches, and modest expenditures (all 
chapters). 
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7 




Software 
Operations 

Reports for 




School 
Improvement 



Even the underlying purpose of the entire project can be expressed 
as "how to improve student learning." The use of clear learning objec- 
tives with Criterion Referenced Tests (CRTs) appears as one avenue to 
be pursued. The Bureau County Model Practices Project discussed in 
this report illustrates the feasibility of meeting the expectations and man- 
dates of the public and their elected representatives. In terms of estab- 
lishing feasibility, the project discussed herein is considered highly suc- 
cessful for the schools in the cooperative. Even more important, many of 
the methods and results probably can be utilized by the majority of school 
districts across Illinois. But most important is the inherent flexibility of the 
methods. That flexibility allows school districts to use any desired "how- 
to" segments without obligation to implement an entirely identical project. 

Tlie offered flexibility is ample evidence that the "how-to" content 
of this report is NOT intended to say "This is the only way." As a reader 
and potential user of this report, you are encouraged to pick and choose 
(and modify) whatever "how-to" chapters are appropriate to your needs. 

The order of presentation of the chapters loosely coincides with the 
sequence of events in the school improvement cycle. Reading them in 
order seems logical and is recommended. On the other hand, reading or 
at least reviewing the final chapters first could help the reader understand 
the target toward which all of the efforts are being directed, namely the 
preparation of reports to guide the improvement of education in the school 
districts. Therefore, Chapter 5 is crucial and does contain information for 
all users of this report. 

Some readers, such as assessment coordinators who have their 
assessment procedures and instruments virtually completed, probably will 
focus also on Chapters 3 and 4 about the use and power of the software 
and hardware. Others, including administrators in small or low-budget 
school districts, could find a valuable and viable model for cooperative 
efforts in Chapters 1 and 2. Anyone unfamiliar with the Illinois mandate 
for educational reform will find an overview in Chapter 1. 
Page 6 l.A How to Read This Report 
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Wherever you place your emphasis, you are encouraged to al least 
browse the other chapters. The Bureau County Model Practices Project 
is an integrated effort. By understanding the entire project, you will be 
able to utilize better the particular aspects that fit your needs. Also, the 
cooperative project is continuing and growing in Bureau County, so the 
personnel there may be contacted for further comments based on their 
ongoing experiences and evolving methods and materials. 
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B Legislative Initiative 
• for School Improvement 



The improvement of education is a major national concern that 
must be addressed at the state and local levels. Illinois is one of forty 
states with current legislated efforts utilizing various forms of Criterion 
Referenced Testing (CRT), Mastery Learning, or Minimum Competency 
Testing. In essence, the development of learning goals, objectives and 
assessments of student progress toward those goals began in Illinois as a 
result of legislation enacted in 1985. 

The Illinois initiative is clearly stated in State Goals for Learning:, 
and Sample Learning Objectives, (page iii) 

The 1985 educational reform legislation addressed nearly 
every aspect of schooling and provided a unique opportunity for 
local school districts and the State Board of Education to work 
cooperatively to improve education in Illinois. One of the most 
important pieces of the legislation, one which has long-range 
implications for learning and teaching in Illinois schools, provides 
for the development of learning goals and assessment systems at 
both the state and local levels. 

Public Act 84-126, effective August 1, 1985, amended The 
School Code of Illinois to include, for the first time in the state's 
history, a definition of schooling and a requirement that the goals 
for learning be identified and assessed .... The legislation 
defines the primary purpose of schooling as the transmission of 
knowledge and culture through which children learn in areas 
necessary to their continuing development: language arts, mathe- 
matics, biological and physical sciences, social sciences, fine arts, 
and physical development and health. 
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According to the schedule for iniplenientaiion, language arts was 
the first learning area to be addressed. The other areas are to be phased 
in over the next several years. The initial state-prepared assessment of 
reading under the new program took place in April of 1988. 'Hie state- 
wide assessments will be conducted each year in April. By May of 1989, 
cv^ry public school district in Illinois is expected to complete locally 
acceptable assessments of language arts in grades 3, 6, 8, and 11. 

The Illinois State Board of Education (ISBH) was instructed to 
establish the goals for learning for cacli area. These have been published 
in six books entitled State Goals for Lcanum'. For laniiuajze arts, six coals 
are specified. Goal 1 is the following (and is u.scd in the examples of this 
report): 

As a result of their schooling, 
students will be able to read, 
comprehend, interpret, evaluate, and 
use written material. 

Each goal is briefly discussed and has a .set of statements of "Gen- 
eral Knowledge/ Skills." Goal 1 has .seven statements (A to G) of knowl- 
edge and .skills, including: 

1C. Reading for various purposes and indication of text to accomplish 
each purpose, (page 6) 

The guidance from the ISBI- goes one step further to provide 
"sample learning objectives" for each of the designated grade levels (3, 6, 
8, and 11). Although these are merely suggested objectives, they arc also 
a guide to what the State might use in its own state-wide Criterion 
Referenced Tests based upon the same main state goals. Fourteen (14) 
.sample learning objectives for Goal 1 are given for the third grade. The 
one used as the example in this report is: 

"By the end of Grade 3. given the readers' prior knowledge and 
reading material with appropriate vocabulary demands, sentence 
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complexity, organizational plan, and concept load, students 
should be able to: 

C2. Use appropriate texts such as fiction, nonfiction, poetry, letters, 
directions, and reference materia! to accomplish the various pur- 
poses for reading." {State Goals, p. 13) 

In summary, the Illinois State Board of Education (ISBH) has pro- 
vided extensive guidance concerning the general goals and specific objec- 
tives of learning. What it has NOT done is prescribe to the schools how 
the learning is to lake place, nor even how such learning is to be assessed 
at the local level. Flex.Dility is abundant within the broad guidelines. 

The ISBE has established a procedure whereby each district must 
specify its locally determined district-wide ^earning objectives and how 
they will be assessed. This is reported to the ISBB through a Learning 
Assessment Plan (LAP). After implementing what was specified in its 
LAP, each district must report the results in a School Improvement Plan 
(SIP). 

The entire LAP/SIP process of "goal-directed" school improvement 
can be viewed as a constructive cycle. Hie basic idea is shown in I-igure 
1-1, but more interaction occurs between the components. Although 
generally applicable to other states, the examples that follow are spec .c 
for the Illinois Goal Assessment Program. 
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Figure 1-1 : Cycle of School Improvement through Goal-Directed Education 



A glance at the cycle shown in Figure 1-1 reveals three key facts., 

1. Each school district must establish district-wide goals and a 
Learning Assessment Plan (LAP) that relates to the State 
Goals (Ovals C and D on Figure 1-1); 

2. Through some combination of guided instruction (Oval E), 
assessment method (Oval F), and obtaining of results (Oval 
G), each school district is to operationalize what was speci- 
fied in the LAP; and 
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3.. The educational outcomes of the above efforts are to be 
incorporated into a School Improvement Plan (Oval H) that 
has impact for improving the District-Wide Learning Objec- 
tives (Oval C) and the Learning Assessment Plan (Oval D). 

Thus the cycle starts over again in an upward spiral with improve- 
ments in objectives, methods, and, most important, student learning. 
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After the legislation was enacted, the Illinois State Board of Educa- 
tion (ISBE) funded twenty-six (26) "Model Practices Projects. . . to 
develop and disseminate effective practices in the Learning Assessment 
and School Improvement Plan process." 

Many of the projects focused on specific tasks such as particular 
learning areas, statistical analyses, or assessment methods. A few pro- 
jects were directed toward the fall scope of the assessment procedures. 
The Bureau County Learning Assessment Cooperative (BCLAC) was 
funded for one of the broadest projects. The full title and short abstract 
are as follows (with emphasis added): 



A Cooperative Approach which Integrates the Scoring of 
Assessment Instruments Providing Exemplary Reporting 
Procedures and Validity and Reliability of Locally Designed 
Testing Instruments 



Abstract: The project generates a generic set of materials for use by 



similar districts in their LAP/SIP process. Trained consul- 
tants deliver the process within the Learning Assessment 
Cooperative and beyond as requested. Phase I focuses on 
technology applications for the development and alignment of 
assessment instruments; phase II focuses on [scoring], 
reporting, materials development, and consultants. 



To supplement the funding provided by the ISBE Department of 
School Improvement Services, the BCLAC pooled the diverse resources 
of the twenty-two school districts in Bureau County to establish a pilot 
project of "model practices" to assist other Illinois school districts in 
meeting legislative mandates of the Illinois General Assembly. 
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The twenty-two school districts in Bureau County range in size 
from 30 to 1,150 students. Of the twenty :wo districts, eleven are ele- 
mentary districts, five are high school districts, and six are K - 12 unit dis- 
tricts. One high school district has students from five separate elementary 
districts feeding into it, while another has three feeder elementary dis- 
tricts. The other three high school districts receive students from single 
elementary districts with the same approximate geographical area. The 
total number of students served by these twenty-two school districts is 
6,600. 

The county's population is primarily rural, with the largest commu- 
nity (Princeton) having a population of 7,300 and the majority of the other 
communities with populations of less than 1,500. Agricultural services 
and light industries provide most of the employment opportunities. Small 
retail businesses and professional services (medical, legal, accounting, 
banking) employ the remaining work force. 

School districts in Bureau County have a long history of cooperative 
activities stimulated by the Bureau County Administrators Association. 
But no prior effort was as extensive as this project which evolved from 
Senate Bill 730. Even the largest school district in Bureau County per- 
ceived major difficulties in developing an assessment system that was 
prescribed by law. 

At the recommendation of Larry Marsh, President of the Adminis- 
trators Association, the districts shifted their focus and utilized the funds 
appropriated for staff development, learning objectives and Title II to 
develop a locally designed criterion referenced assessment system that 
would match the State Goals for Learning and Sample Learning Objec- 
tives published by ISBE. Marsh was appointed administrative agent and 
coordinator of the cooperative. Approximately 70% of the twenty-two 
school districts joined the Cooperative initially, with the remaining districts 
joining as deadlines drew nearer for the submission of Learning Assess- 
ment Plans and as they reali 3d that generic standardized assessment 
instruments did not assess all the specified minimal objectives that had 
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been adopted. Furthermore, even the largest districts in the county came 
to the realization that cost factors prohibit development of an adequate 
learning assessment system alone. 



In November of 1985, the administrators infonnally formed the 
Bureau County Learning Assessment Cooperative. Funding came from 
the Staff Development, Learning Objectives and Title 2 funds provided by 
Senate Bill 730 to each district. 

The contributions to develop a locally designed system ranged from 
$150.00 to $600.00 for the smaller districts to $2600.00 to $4200.00 for the 
larger districts in the county. Cooperative efforts wouM have generated 
$30,000.00 had the Staff Development, Learner Objectives, and Title II 
appropriations been fully funded by the state government. However, 
reductions in appropriations reduced these combined totals for the Bureau 
County Cooperative to approximately $20,000.00 annually. 

After receiving the "Model Practices" grant to develop the prod- 
ucts discussed in this report, a steering committee was appointed to assist 
the coordinator. The steering committee functioned much the same as 
any executive committee in cooperative agreements. The steering com- 
mittee was authorized to develop or purchase the necessary ^ivvare and 
hardware for scoring and reporting. 

An extremely important aspect of the project was the cooperation 
by the districts and their staff. As with most successful voluntary cooper- 
ative organizations, the willingness and ability of the leaders to lead must 
be matched by the encouragement and support of all others who partici- 
pate and stand to benefit. Strong leadership and strong support con- 
tributed to the success of the Bureau County model practices project. 
Any attempts to replicate the organizational aspects of this project must 
pay attention to these issues. For further information, contact any mem- 
ber of the steering committee, as named at the end of this publication. 
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The full project of the BCLAC has completed its initial phases. 
Those efforts focused upon the development of the cooperative's language 
arts Learning Assessment Plan (LAP), language arts curriculum guide, 
reading assessment instruments, and a system of machine-assisted scor- 
ing with exemplary reports. Significant progress has been made in the 
follov/ing seven major elements of the Bureau County Model Practices 
Project for school improvement: 

1. Develop school district LEARNING ASSESSMENT 
PLANS (LAPs) in the learning area of language arts; 

2. Develop, disseminate, and implement county-wide 
CURRICULUM GUIDES for language arts for grade 
levels 3, 6, and 8, consistent with the learning assess- 
ment plan; 

3. Develop, pilot, evaluate, refine, and administer a local 
assessment system that includes both an ITEM TEST 
BANK organized by learner objectives and published 
ASSESSMENT BOOKLETS; 

4. Develop or purchase COMPUTER SOFTWARE to 
serve as an appropriate "scoring template" including 
determination of INDICES OF VALIDITY, RELIA- 
BILITY, DIFFICULTY AND DISCRIMINATION in 
ways that can be replicated by other school districts; 

5. RECOMMEND HARDWARE AND SOFTWARE 
MATERIALS to assist other school districts in the 
se ection, cost analysis, and use of necessary materials 
to complete the assessments; 
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6. Generate reports for administrators, teachers and par- 
ents via EXEMPLARY REPORTING PROCEDURES 
for purposes of SCHOOL IMPROVEMENT based on 
the results of the developed local assessments, includ- 
ing assessment results for individual remediation pur- 
poses and item analysis; 

7. Identify and/or prepare CONSULTANTS AND 
WRITTEN MATERIALS to assist other districts in 
using the materials. 

The first three elem.ents above are objective-related tasks in the 
domain of the curriculum specialists. The Bureau County experiences 
with those tasks are presented in Chapter 2. The experiences serve as 
one example, not as a formula for all school districts. 

Elements Nos. 4 and 5 above are covered in Chapter 3 about the 
computer software and hardware. The "how to run the software" dis- 
cussion in Chapter 4 is part of the written materials mentioned in elemeni 
7. Element 6 for the exemplary reports plus the indices in element 4 are 
presented in detail in Chapter 5. 

When taken together, the chapters of this publication provide an 
overview of how these seven elements were interwoven and accom- 
plished in Bureau County prior to August 1988. This publication is 
intended not merely to report about the project, but also to serve as a 
pragmatic and flexible guide for other school districts that have agendas 
that contain similar elements. 
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Specific Learning 
Objectives and LAP 



The preceding chapter shows how the momentum 
has already begun through the cycle of school improvement. 
Much guidance from the ISBE has been made available 
about the goals and learning objectives. At the local level, 
district-wide learning objectives have been formulated, most 
programatically expressed as the Learning Assessment 
Plan (LAP). Each district's LAP must include pages such 
as the one in Figure 2-1, but the contents may include 
whatever learning objectives the district chooses and can 
justify. Each LAP is essentially an outline stating the school 
district's intentions. Therefore, quality in the LAP will pay 
dividends at subsequent stages of the cycle. 

As an example of "how to" complete the LAP, the 
Bureau County Learning Assessment Cooperative 
(BCLAC) employed consultants from Bradley University to 
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assist in developing the Learning Assessment Plans for the districts. 
Committees of teachers representing all the districts were organized to 
formulate the detailed plans to match or relate to the goals and objectives 
set forth by the ISBE's State Goals for Learning. 

One committee was established to formulate the objectives for 
Reading and Language Arts. Those objectives (see Figure 2-2) were 
included on the Learning Assessment Plans (LAPs) that were submitted 
to the Illinois State Board of Education. LAP Committee members were 
appointed by the BCLAC coordinator 'vith the assistance of the superin- 
tendents of individual school districts within Bureau County. One of the 
primary goals was to appoint committee members who could represent 
each grade level affected (3, 6, 8, and 11) and as many of the school dis- 
tricts as possible in Bureau County. Approximately fifty percent of the 
school districts had representatives assigned to the committee to develop 
the Learning Assessment Plans. The four or five members for each 
grade level are named in the Acknowledgements. 
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1. Use appropriate texts as fution. nonfiftuxj. poetry, letters, 

directions, and rt»ference material to accomplish the various purposes 
for reading. 
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Subgroup # LAP # Description 

1 lA 1 Locate information that is explicitly stated in the text, 

2 IA2 Summarize the important ideas of the text and the 

important supporting details. 

3 IB 1 Ask questions and make predictions alx)ut a passage 

prior to reading. 

4 IB2 Ask questions and make predictions about a passage 

while reading. 

5 IB3 Ask questions after reading that take into account the 

entire text read. 

6 ICl Use appropriate texts such as fiction, nonfiction, 

poetry, letter, directions, and reference material to 
accomiplish the various purposes for reading. 

7 ICl Understand the difficulties of the text, rajuirements 

of the task, and their own knowledge, abilities and 
motivation. 

8 lEl Make inferences about the text such as unknown 

vocabulary, casual relationships, author's purpose, 
characters' emotions and motives, mood and tone 
using information from the !ext and prior knowledge. 

9 IFl Use, synthesize and analyze infonnation from a 

variety of sources to enhance understanding, to 
compare/contrast, to verify information, and to 
expand knowledge. 

'0 IG 1 Explain and verify answers to questions abcMit what 

has been read. 



Figure 2-2: Assessment Subgroup Numbers and Learning Objectives Used 
by BCLACin Spring 1988 for Third Grade Language Ans 
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• Curriculum Guide 



Tlie sending of the Learning Assessment Plan to the ISBR signifies 
that the school board-, administrators, and leading teachers have estab- 
lished the directions and pathways to follow. Now the task is to accom- 
plish the goals through the organized and coordinated efforts of dozens of 
teachers and hundreds or thousands of students. A Curriculum Guide 
becomes essential. 

Tlie Curriculum Guide serves two major functions: to communicate 
to the teachers precisely what the school district expects to be taught/ 
learned in the classroom, and to suggest instructional sequences to 
achieve district goals by the end of each giade level, not just for grades 3. 
6. 8, and 11. In essence, the Curriculum Guide provides assistance and 
clearly stated expectations for the teachers to use with their daily lesson 
plans. 

In Bureau County, the two university consultants worked with eight 
teachers appointed to the Curriculum Guide Committee by the BCLAC 
coordinator in consultation with the superintendents of the cooperating 
districts. The Curriculum Guide Committee developed the K-8 curriculum 
guide and is still developing the 9-12 learning area guides for Language 
Arts. The guides are designed to match the objectives specified in the 
LAP as well as other grade levels not included in the LAP. This com- 
mittee analyzed numerous sample guides from publishing companies, 
other districts, and other state departments of education. The guide was 
designed for vertical articulation through the grades and horizontal articu- 
lation across the learning areas. A full set of examples from the final 
Curriculum Guide is reprinted on the next seven pages. For the examples 
given, the selected learning objective is to "Use appropriate . . . reference 
material to accomplish the various purposes for reading." Specific "how 
to do it" comments are given in Section D of this chapter. 
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Preface and State Goals 

In 1987, the 22 Bureau County school districts established the Bureau County 
Learner Assessment Cooperative (BCLAC) for the purpose of improving and coordinating 
educational programming throughout the county. In compliance with the 1985 Educational 
Reform Legislation, Public Act 84-126, the BCLAC Language Ans Curriculum Advisory 
Committee established language ans goals that are consistent the primary purpose of 
schooling. The State Board of Education defines the primary purpose of schooling as "the 
transmission of knowledge and culture through which children learn in areas necessary to 
their continuing development." 

In order to meet the state mandates, the BCLAC membership reviewed and revised 
previously adopted mastery level language ans objectives for grades K-12, 
Representatives from the 22 panicipating school districts finalized this review. The ISBE 
State Goals for Language Arts Grades 3, 6, 8, and 1 1 were reviewed. The General 
Knowledge/ Skills for each grade were reviewed for inclusion. Curriculum 
recommendations for grades 9-12 are included in a separate document. 

The following State Goals for Learning in Language Ans are essential for student 
success in virtually all areas of curriculum. The State Board concluded that language ans 
develops clear expression and critical thinking through the study of literature and the 
development of skills in reading, writing, speaking, and listening. 

As a result of their schooling, students will be able to: 

• read, comprehend, interpret, evaluate, and use written material; 

• listen critically and analytically; 

• write standard English in a grammatical, well-organized, and coherent 
manner for a variety of purposes; 

• use spoken language effectively in formal and informal situations to 
communicate ideas and information, and to ask and answer questions; 

• understand the various foms of significant literature representative of 
different cultures, eras, and ideas; 

• understand how and why language functions and evolves. 

BCLAC LANGUAGE ARTS 
CURRICULUM ADVISORY COMMITTEE 

AUGUST 1988 
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GENERAL INFORMATION 
(for K-8 Language Arts Curriculum Guide) 

Grade Level Color-Code 

KINDERGARTEN - Goldenrod 

FIRST - Salmon 

SECOND - Yellow 

THIRD - Green 

FOURTH - Red 

FIFTH Pink 

SIXTH - Lavender 

SEVENTH -Blue 

EIGHTH - Buff 

Glossary of Terms 

appropriate complexit> - Each item on the matrix has been selected according to its 
appropriate level of complexity; - that students are able to accomplish the designated skill 
which is acceptable at that level of development. 

alliteration - The use of the same sound at the beginning of stressed words in a group or 
line of verse, as the/s in the phrase "a fair field full of folk." 

analogy - A likeness that exists between two objects that are in other rc^pe^ts not the 
same: There is an analogy between the wings of a bird and the wings of an insect. 

base word (root word) - In grammar, a word to which affixes and inflectional endings 
may be added. 

blend - A blend is two consonant letters blending together, each retaining its own sound. 

declarative sentence - Making a statement 'Til be home at five o'clock." is a declarative 
sentence. 

demonstrative pronoun - Serving to point out a person or thing. In the sentence, "This 
is my book*', this is a demonstrative pronoun. 

digraph (vowel) - A combination of two vowels having one sound, as oa in boat. 

essay - A short composition in which the writer gives his or her own ideas on a single 
subject. 

familial - Of, characteristic of, or having to do with a family. 

grapheme - The set of units of a writing system (as letters and letter combinations) that 
represent a phoneme. 

homophone (homonym) - Any of two or more words that are pronounced alike but have 
different meanings, origins, and usually spellings, as son and sun. 

hyperbole - An obviously exaggerated statement made for dramatic eifect, as in 'The 
coach is as tough as nails." 
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idiom - An expression having a special meaning different from the usual meaning of the 
words. "To put up with" is an idiom meaning "to tolerate or endure." 

imperative sentence - Having to do with, or being the mood or a verb that expresses a 
command, request, or plea, (e.g.. Go at once!) 

indeflnite pronoun - A pronoun, as one, any, or few, that does not identify or limit the 
person, persons, thing, or things to which it refers. 

interrogative sentence - A sentence that asks a question. 

metaphor - A figure of speech which suggests, without saying so, that one thing is like 
another. It applies a word or a phrase to something to which it does not ordinarily apply, 
as in "The room was a beehive of activity." 

N.A. - Infonnation that is not applicable at this grade level. 

personification - A figure of speech in which inanimate objects or qualities are spoken of 
as having human characteristics, as in the saying "Necessity is the mother of invention." 

phoneme - One of a set of the smallest units of sound serving to distinguish one word 
from another. The words pin and tin are distinguished by the phonemes p and f. 

phonogram - A sign or symbol representing a word, syllable, or sound. 

proper adjective - An adjective formed from a proper noun, (e.g. Swiss as in Swiss 
cheese). 

root word - see base word. 

simile - A figure of speech in which one thing is compared to another that is different in 
many ways, by the use of as or like, (e.g.. The professor is as wise as an owl.) 

theme - A subject or topic of discussion; a short essay. 
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MASTERY LEVEL OBJECTIVES BY GRADE LEVEL 

GR. Read- Phon. Comp. Voc. Rdg. Stdy. List. Spkg. Wrth. Lit. TOTAL 
iness Var.Pur. Skis. Appr. 





(K-3) 


(K-8) 


(K-8) 


(K-8) 


(3-8) 


(K-8) 


(K-8) 


(K-8) 


(K-8) 


(K-8) 




K 


9 


0 


3 


0 




0 


0 


0 


0 


2 


(14) 


1. 


3 


3 


1 


0 




1 


5 


1 


6 


3 


(23) 


2. 


1 


4 


0 


2 


~ 


0 


1 


2 


3 


5 


(18) 


3. 


1 


9 


2 


2 


0 


2 


0 


2 


9 


1 


(28) 


4. 


- 


6 


2 


1 


0 


1 


0 


0 


4 


2 


(16) 


5. 




0 


3 


2 


0 


0 


2 


1 


8 


5 


(21) 


6. 




0 


6 


3 


0 


2 


0 


0 


18 


1 


(30) 


7. 




0 


3 


0 


0 


3 


3 


0 


16 


2 


(27) 


8. 




0 


5 


1 


1 


1 


0 


0 


14 


0 


(22) 


OTAL 


14 


22 


25 


11 


1 


10 


11 


6 


78 


21 


(199) 
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EXPLANATION OF GRADE LEVEL OBJECTIVES 

For the teacher's convenience, the objectives for each grade level 
have been cross-referenced with the Language Arts Curriculum Content 
Objectives found in the matrix. The numbers and letters found in each 
section for this grade level are the same numbers and letters that appear 
in each section of the matrix. For example, at each grcde level 

11. Phonics 

B. Vowels 

3. Identify and articulate long vowel sounds 
b. vowel combinations 

will be the same as II.B.3.b. of the matrix: 

The following terms and letters are used to indicate level of mastery 
of the objectives at each grade level in the same manner that they are 
used in the matrix.. 

I - Introduce - is defined as formal presentation of material 

for the first time; 

D - Develop/Maintain- is defined as expanding upon previously 

introduced material as a sequential step to 
mastery of the area; 

is defined as restating material which has 
been mastered previously for the purpose of 
reinforcement of the material. 

is a list of ISBE Language Arts Goals and 
Knowledge/Skills that correlate with the 
respective BCLAC instructional objectives. 



R - Review - 



Goal & K/S - 
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BCLAC 

MASTERY LEVEL LANGUAGE ARTS OBJECTIVES 

GRADE 3 



VI. 


Study Skills 


Introduce 


Develop/Maintain 


Master 


Review 


1. 


Use basic location aides 

a. abc order 

b. table of contents 

c. textbook indexes 


X 




X 
X 




Z. 


Locate and interpret graphic aides 




X 






3. 


Locate and interpret information in 
reterence sources 

a. dictionary/glossary 

b. encyclopedia 

c. thesaurus 


X 
X 


X 






4. 


Identify and locate parts of a book 




X 






r 
J. 


Distinguish between types of books 
(fiction, nonfiction, references, poetry) 




X 






D. 


Use sources of information 

(e.g., schedules, newspapers, telephone book) 




X 






7. 


TTt*1 1*11/ 1 t 

Use library skills (e.g., card catalog, 

Dewey Decimal System, computerized on-line 

catalog) 


X 








8. 


Use basic survival skills (e.g., fill out forms, 
take messages) 


X 








9. 


Use research skills(e.g., notetaking, 
outlining, summarizing, synthesizing) 


X 









* Objectives not covered in the adopted reading/grammar series. 
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• Assessment Items 




Members of the LAP and Curriculum Guide Committees served on 
the committee that developed the assessment instruments. This commit- 
tee analyzed standardized tests, textbook tests, and teacher-made instru- 
ments. Five to ten items were developed to match each objective that 
had been filed on the LAP. These items were matched to each of the 
subgroups identified for each grade level. 

All committees met monthly for a minimum of nine day-long meet- 
ings during the school year. Expenditures for travel and catered meals 
were charged to the cooperative's budget. The districts which had 
teachers serving on committees were reimbursed for the hired substitutes. 
Those districts incurred additional expenditures of at least $360.00 per 
teacher. 

Upon completion of a sufficient bank of assessment items, pilot 
testing of students in grades 3, 6, and 8 was done at selected districts 
throughout Bureau County during the early part of the second semester of 
the 1987-88 school year. Pilot assessment of writing skills was also done 
for grades 3, 6, 8 and 11, utilizing the assessment guidelines established in 
the Illinois Writing Plan. 



Manual scoring and close observation of students are often appro- 
priate during pilot testing. However, the advantages offered by 
machine-assisted scoring should not be overlooked. For example, 
the reports and statistics described in Chapter 5 can be of great 
value for evaluating the quality of the pilot questions separately and 
how they relate together to each learning objective. 

The results of the pilot assessments were utilized to determine 
content validity. A committee of teachers was selected from the partici- 
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pating districts to serve as content specialists for selected grade levels. 
The committee, in conjunction with the curriculum consultants from 
Bradley University, also contributed to the Learning Assessment Plans 
and Curriculum Guide. An "Examiner's Manual" was prepared to 
enhance the uniformity of administration of the assessment instruments. It 
serves as a "how to do it" guide and is reprinted in its entirety. With 
appropriate local modifications, any school district is welcome to use it. 

After reviewing the piloted items for validity and reliability (as dis- 
cussed in Chapter 5), items were selected for inclusion in the final form of 
the assessment instruments based on acceptable difficulty levels. A 
criterion level (minimum percentage of answers to be correct) of sixty 
percent (60%) was agreed to by the participating distrcts as a county- 
wide minimum. Also, an expectation level (minimum percentage of stu- 
dents expected to attain the criterion level) was set at sixty percent 
(60%). Each school district in the cooperative has the right to set higher 
criterion levels and expectation ]evels in all or selected learning objec- 
tives. The criterion and expectation levels are set at the beginning of the 
cycle when the Learning Assessnicnt Plans (LAPs) are submitted to the 
State Board of Education. An important part of the analysis of results for 
school improvement should relate to these minimum targets. 

The process involving objectives, questions, pilot, and analysis cul- 
minates in the preparation of final assessment instruments that have con- 
tinuity with the planned curriculum, providing an assessment system valid 
for its intended purpose. Examples of test items from the bank as they 
appeared on the finished third grade assessment instrument are shown 
after the Examiner's Manual. 
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Bureau County Learner Assessment Cooperative 
Reading Criterion-Referenced Test 
Examiner's Manual 

PURPOSE 

The Bureau County Learner Assessment Cooperative (BCLAC) basic 
skills reading test has been developed to survey students' academi 
achievement for Grades 3, 6, and 8. The test was designed to 
correlate with the goals and objectives included in the Bureau 
County Learning Assessment P2ans as mandated by the State of 
Illinois. The test results will provide information to assist the 
school district in 

.•..planning and evaluating programs 
....assessing students mastery of state objectives 
....identifying needs of individual students 
comparing individual and group growth annually 

GENERAL INFORMATION 

This test was developed by a committee of reading teachers and 
piloted by a random sampling of students in Bureau County. The 
test was designed to be administered in the spring. 

MATERIALS 

All students will need the following materials: 

....a test booklet 

. . . .answer sheet 

....2 sharpened #2 pencils 

. . . .eraser 

....place markers (optional) 
PLANNING FOR THE T EST 

The test should not be administered immediately before or after a 
vacation break or holiday. Two or three days before the test, you 
should discuss with the students plans for administering the test. 
Also, a letter should be sent home to parents, informing them of 
the testing schedule and soliciting their support for maximum 
student performance (i.e., suggest that students get an adequate 
number of hours of sleep the night before the test and a good 
breakfast the morning of the test). Examiners should become 
acquainted with the testing material before the test is given. Tt 
is important for examiners to be careful to not provide clues to 
correct answers. 



Examiner's Manual 
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TIMING 

These tests are not timed. Once t^e actual testing begins, it is 
suggested that students should be allowed to work for an 
approximate time period of one hour. The students should be given 
a break period and then continue to work until the test is 
completed. The estimated time to complete the testing session, 
including distribution and collection of mareriaj-S, examiner's 
instructions, and a fifteen minute bre£.k, is approximately two to 
three hours • 

SPECIAL CONSIDERATION 

Students who have been identified and receive Special Education 
services may take the tests with their appropriate grade placement* 
At zhe discretion of the local district, special education students 
may be excluded from the district results. Upon completion of the 
tests, the answer sheets of those students that v/ill not be 
included in the district results should be removed from the class 
set and scored separately. 

GENERAL INSTRUCTIONS 

1. Students will not write in the test booklets. No marks should 
be made on the answer sheet, except the answer responses. 

2. Identifying information should be completed on the answer 
sheet, including name, grade, I.D. number and special code for 
Special Education. (Please see the attached sample answer 
sheet.) This information will be completed by the teacher for 
grade 3. 

3. Call students' attention to the fact that answer spaces are 
given for A, B, C, D, and E (but D and E will not always be 
used) . 

4. Directions to be read aloud to students will appear in boldface 
print and in a box in the Examiner's Manual . 

COMPLETING THE NAME AND I,D. GRIDS 



Have the students fill in the Name and I.D. grids on the answer 

sheet, unless it has been done in advance for them. Use the 

follov/ing codes for completing the I,D, grid. Code only the 
student and I*D, grids. 
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1. Use columns A, B, and C for the school district code: 

School ABC 

Cherry Grade School Dist, #92 092 

Dalzell Grade School Dist. #98 098 

DePue Unit School Dist, #103 103 

Hall High School Dist, #502 502 

K-'.sbeer Grade School Dist. #23 023 

Ladd Grade School Dist. #94 094 

LaMoille Comm- Unit Dist. #303 303 

Leepertown Grade School Dist, #175 175 

Maiden Grade School Dist, #84 084 

Manlius Comm- Unit Dist- #305 305 

Neponset Comm- Cons- Dist- #307 307 

Ohio Grade School Dist- #17 017 

Ohio High School Dist. #505 505 

Princeton Elementary Dist- #115 115 

Princeton High School Dist- #500 500 

Spring Valley Elementary Dist- #99 099 

Tiskilwa Co^m. Unit Dist- #300 300 

Walnut Grade School Dist- #285 285 

Walnut High School Dist- #508 508 

Western Comm- Unit School Dist- #306 306 

Wydnet Grade School Dist- #126 126 

Wyanet High School Dist- #510 510 

?• Use column H for the teacher code- Each school district 
should designate this code. Begin with "1" for the first 
teacher in the given grade and proceed in ascending order 
until all teachers in the grade for the entire school 
district have received a unique code- 

For example, American Elementary District No. 123 has two 
buildings with third grade classes- There are two third 
grade teachers in one building and three third grade 
teachers in the other building- Therefore, the teachers 
would be designated codes 1 through 5- 

3- Use columns I and J for the student code- Each student in 
a class should be designated a code. Begin with "0" and 
proceed in ascending order ^^ntil all students in the class 

' ^ ^ ^ * * - ^ ^ >- • 

Code "Special Education" students with the highest numbers 
used- Separate their answer sheets for scoring purposes 
from the rest of the class after the test has been 
completed . 
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For example, Mrs. Smith's third grade class has 25 
students. Two of these students have been determined to 
be LD students and have lEP's. Begin with student one as 
"01" and proceed through "25." The special education 
students will be coded "24" and "25" respectively. 

PLEASE SEE THE ATTACHED SAMPLE SHEET. 

INTRODUCING THE TESTS 

Read the follcwing aloud to the students: 



Make no marks in your booklets. They must be used again by 
other students. Don't open your test booklet until I tell you. Be 
careful not to fold or wrinkle your answer sheet or bend the 
corners. Keep it as clean as you can. 

If you want to change a mark, make a new mark and then erase 
your first mark completely. 

Make sure your answer sheet is on side 1. 



Hold up an answer sheet and demonstrate. 
When students have finished, continue: 



Are there any questions? 

Today you will be taking a reading test that will tell how 
well you remember and understand what you have learned. These 
tests will not affect your grades, but will help identify your 
strengths and weaknesses. It is important that you do your best. 
Answer each question as well as you can. If you are not sure of 
the correct answer, pick the one you feel would be the best 
choice* A good guess is better than no answer at all, and you will » 
not be penalized for incorrect answers. j 

Now look at the sample on Page 1 of the Test Booklet. ' 



Give the students time to look at the test booklet to see the 
sample on page ]. Check to see that all students are on the 
appropriate page. 



Read the following paragraph and the sample question. Choose 
the best answer. (respond aloud). The correct answer is "C," so 
you would fill in the rect tangle "C" on your ansv/er sheet if this 
were an actual question. 

Mark only one answer for each question and double check to make 
sure that you have filled in the circle you meant to fill in. 

You will begin to work when I say, "Begin," and continue to 
work until you come to the word "Stop." 

Are there any question? 

Examiner's IVIanual 
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1. You rray read aloud the directions located in the test 
booklet if you feel it is appropriate* 

?• The test for grade 3 has the word "Stop" included at 
various places in the test* This will give slower 
students an opportunity to catch up. Use your 
discretion to decide when students should continue to 
the next section (s)* 



THIRD GRADE SAMPLE 



There are about 2, 500 kinds of snakes* Some are bj.g enough to 
swallow a grown-up deer whole* Some snakes in Africa can kil] a 
man in a minute! But most of the snakes we see are harmless. 

Nearly all snakes are ready to take care of themselves v;hen 
they are born. A baby rattlesnake two minutes old can coil and 
strike. And he is poisonous, too. 

A snake never stops growing. This is why he has to change his 
skin often. He gets a brand new suit about every six weeks. Some 
snakes have lived as long as 23 years. 

Every six weeks a snake: 

A. turns inside out 
strikes 

C. loses his old skin 

D. eats 



SAMPLE - GRADES SIX AND EIGHT 



You need energy for everything you do. It takes energy to 
smile, to frown, and even to y^wn. You get the energy you need 
from the food you eat. A locomotive also needs energy to make it 
run. It gets this energy from oil or coal. Most of the energy 
used in the world starts from the sun. 



]. Most of the world's energy starts from: 

A. three different places 

B. the moon 

C. the sun 

D. coal and oil 

When the sample question has been completed, ask: 



"Are there any question?" 



DIRECTIONS FOR CONCLUSION OF TESTING 

When the test has been completed, please complete the following 
items: 



1. Immediately collect the student test booklets, answer sheets, 
pencils, and scratch paper. 
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2, If special education students have been tested, separate- their 
answer sheets into a separate set for scorifig purposes, 

3, Discard the scratch paper, 

4, Check the answer sheets for correct marking of the answers. 
Darken the students' answer choires and erase ai.y stray nark?, 
if needed. 

5, Return all test booklets, answer sheets, and pencils to the 
designated persons . 



Page 38 Examiner's Manual 



Sample Assessment Items 



BUREAU 
COUNTY 

Criterion - Referenced 
Reading Test 

GRADE 3 



SAMPLE - GRADE THREE 

There are about 2,500 kinds of snakes. Some are big enough to swallow a 
grown-up deer whole. Some snakes in Africa can kill a man in a minute! But 
most of the snakes we see are harmless. 

Nearly all snakes are ready to take care of themselves when they are bom. 
A baby rattlesnake two minutes old can coil and strike. And he is poisonous, 
too. 

A snake never stops growing. This is why he has to change his skin often.. 
He gets a brand new suit about every six weeks Some snakes have lived as 
long as 23 years. 

Every six weeks a snake: 

A. turns inside out 

B. strikes 

C. loses his old skin 

D. eats 
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BUREAU COUNTY 
READING TEST - CRAPE Tl IREE 



Questions 16-22 ask you to select the 
correct reference source that would 
help answer these questions. 

16. Which of the following books 
would you use to write a report about 
Abraham Lincoln? 

A. A dictionary 

B. An atlas 

C. A biography 

D. A book of poetry 

17. Which of the following books 
might be considered fiction? 

A. All About Whales 

B. The Life and Times of Daniel 
Boone 

C. You and Your Body 

D. , The Velveteen Rabbit 

18. Which of the following books 
would contain rhyming words? 

A. Mother Goose 

B. Tom Sawyer 

C C harlottes's Web 

D. Christopher Columbus 

19. Which of the following sources 
would you use to find a map of the 
United States? 

A. Dictionary 

B. Atlas 

C. Table of Contents 

D. Card Catalog 
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20. Which of the following sources 
would you use to find the meaning of 
a word? 

A. ; Dictionary 

B. Atlas 

C. Table of Contents 

D. Card Catalog 

21. Which of the following sources 
would you use to find a book about 
stars and planets? 

A. Dictionary 

B. Atlas 

C. Table of Contents 
O. Card Catalog 

22., Which of the following sources 
would you use to find the beginning 
page of a story? 

A. Dictionary 

B. Atlas 

C. Table of Contents 

D. Card Catalog 

(GO ON TO THE NEXT PAGE) 



2.C Assessment Items 



Sample Assessment Items 



BUREAU COUNTY 
READINC TRST - GRADF. THRRF. 



QUESTIONS 23-25 ASK ABOUT THE 
FOLLOWING TABLE OF CONTENTS. 



X\BLE OF CONTENTS 




TITLE 


AUTHOR PAGE 


Mary and Her 

LdlllU 


M. Goose 


4 


Bees Are Not 
Supposed to Fly 


T Morgan 


11 


The Lion and 
the Mouse 


S. Brown 


17 


Billy and His 
Bike 


R Sandberg 


23 


The Rabbit and 
the Grasshopper 


j. Thompson 


32 



23. Who wrote the story that begins on 
Page 11? 

A. M. Goose 

B. S. Brown 

C. J.Thompson 

D. T Morgan 



24. What is the title of the story written 
by P. Sandberg? 

A. 'The Lion and the Mouse" 

B. . "Billy and His Bike" 

C. "Mary and Her Lamb" 

D. "Bees Are Not Supposed to Hy" 

25. The story "Mary and Her Lamb" 
begins on what page? 

A. 4 

B. 11 

C. 17 

D. 23 
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Notes on How to 
Prepare the Assessment 
• Instruments and Costs 



V 



Although the previous pages of examples are valuable for showing 
"how to do it," they might give the wrong impression. The Bureau 
County Learning Assessment Cooperative did NOT set out to establish a 
model of content nor of presentation style. The content of any curriculum 
guide is highly individual for each school district or cooperative. Also, 
presentation style is only a reflection of the chosen page layout and a 
fancy word processor. But the BCLAC experience DOES OFFER sev- 
eral valuable "how to" insights to the process of implementing the LAP, 
especially in the context of small school districts. 

1. Plan ahead!! As time runs r t, the options diminish. 

2. Any assessment instrument is specific to a set of learning 
objectives or segments of the subject matter. When the 
Bureau County Cooperative reviewed its objectives, the 
administrators and committees soon realized they were not 
satisfied with any previously existing assessment package. 
The questions and scope available did not match those 
desired by Bureau County. This is likely to occur for every 
school district unless the district adopts a pre-set package of 
learning objectives, curriculum guid .tudy materials, and 
assessment items. 

3. The curriculum guides must be developed with great care. 
Inadequacies in the guidelines can lead to unfairness in the 
final assessment and ultimate failure or irrelevance of the 
entire goal-directed school improvement process. 
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4. The actual assessment instruments must be prepared to 
reflect the goals and objecives in the curriculum guide. 
Content of test booklets is far more important than pretty 
printing. Pilot studies, statistical analyses, content validity 
checks and other measures are needed for each variation of 
a test, whether it is to be used by 30 or 300 or 3,000 or 
30,000 students. 

5. Machine-assisted scoring and reporting are essential, even 
with small classes, in order to obtain the necessary statistical 
data to justify confidence in the assessment instrument and to 
provide needed reports for administrators, teachers, students 
and parents. (Further discussion is in the next chapters.) 

6. Small school districts should not attempt such a project alone. 
Even medium-size districts are well advised to enter cooper- 
ative efforts. Cooperation is vital for attaining sufficient 
resources. The essential quality of content, style, 
dissemination, and coordination into the classrooms cannot be 
accomplished on a shoestring budget. 

7. The time and resources (excluding hardware and software) 
needed to prepare a valid and reliable assessment instrument 
for one subject area is at least 12 months at a cost of $30,000 
and 540 person-hours for teacher committees. This essen- 
tially equals one full-time person plus clerical and committee 
assistance. 
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Estimated costs for another school district or group to replicate the 
Bureau County materials for three grade levels for language arts (or for 
three grades in another learning area) are as follows: 



These estimates are for the first-time use with approximately 500 
students tested in each of three grade levels. In this case, the cost was 
approximately $20 per student tested, excluding scoring and reporting. 
Each subsequent year for the same learning area is estimated to cost $4 
to $6 per student tested. If the number of students served by the efforts 
of a cooperative (or large district) were double those of the Bureau 
County Project, the costs might drop to $12 to $15 per student for a first- 
time assessment, with $3 to $5 costs per student for subsequent years. 

Although the gross numbers at first might appear high, the expen- 
ditures in dollars and hours can be considered very reasonable or even 
low-cost investments in light of the benefits to the students, teachers, dis- 
tricts, and state. The key to those benefits is the actual obtaining and use 
of meaningful results for school improvement. The following chapters 
focus on those results. 



Although the preparation of an assessment instrument can be a dis- 
tinctly individual effort of each school district or cooperative, certain pro- 
cedures are more appropriate than others. In addition to the comments 
offered in this publication, at least two other references are highly appro- 
priate for schools in Illinois. Assessment Handbook (1988) and the 1988 
report by Springfield Public School District 186 are quite useful. By saying 



Consultants (external) and coordinators (internal) 
Faculty time (includes substitutes at $40 per day) 



$19,000 
5,000 
1,500 
4,500 



Pilot 
Printing 



Total 



$30,000 



Page 44 



2.D Notes on "How to Do It" 



similar things in different ways, these three totally independent publica- 
tions sponsored by the Illinois State Board of Education are mutually sup- 
portive. Each report has different examples and emphases, but most of 
the methods can be freely intermixed to serve the needs of other districts. 
Even the computer-assisted test scoring systems used by Springfield and 
Bureau County have many similarities (and a few important differences), 
as discussed in Section 3.B.I. In general, in the realm of criterion refer- 
enced tests (CRTs) at least one methodology typified by the Bureau 
County and Springfield experiences does appear to be highly applicable to 
school improvement in Illinois. 
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People, 
Software 
and Hardware 



A 



People First 



Bureau County selected its machine-assisted scoring 
system the right way. People are first, software is second, 
and hardware is third. The first considerations should 
always be the needs and how the people can meet those 
needs. 

Statistical analyses and numerous reports are clearly 
needed; manual processing is out of the question except for 
specialty items. The reports presented in Chapter 5 are a 
good indication of the variety of needs to be fulfilled.. Item 
analysis, validity checks, subtotals for each learning objec- 
tive and data-intensive analyses make evident the need for 
computer-assisted scoring to meet the Illinois school 
improvement mandate. 

Furthermore, whatever solution would be eventually 
selected must be complete, efficient, fast, and user friendly. 



3.A People, Software, and Hardware 




Legislation Lm. Coals 




Objectives 



SIP 




Results^ 




Guide 





Software 
Operations 

Reports for 




School 
Improvement 
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Section 3.C and Chapter 4 give good examples of people-related feature? 
to be expected in hardware and software. 

The solution to the needs also must be affordable. In the initial con- 
siderations by the BCLAC, "affordable" was defir.ed in the budget as 
approximately $10,000 to sen'e the 22 school districts with nearly 7,000 
students. In the final analysis, the BCLAC found that much less money 
could acquire the needed software and hardware, as is summarized in 
Section 3.D. 
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7. The Selection Process 

The second consideration is to select and obtain software that ful- 
fills the needs of the people in the school district. There are literally 
dozens of test scoring software packages available, but only a handful can 
be considered to be of competitive commercial quality. System design, 
capabilities, special functions, versatility, speed, and ease of use are the 
distinguishing features of commercially viable software. Ironically, such 
features combine to make the software seem simple. Perhaps that is why 
so many people and organizations are tempted to write their own pro- 
grams. 

A good comparative illustration is the development of word proces- 
sors that started as costly dedicated machines and software. With the 
advent of microcomputers, hundreds of word processor programs were 
written; some were and are siill given away free. But fewer than ten full- 
featured, commercial-quality word processor programs now dominate the 
micrccomputer scene. Today, nobody would write a new word processor 
progr?m without major financial backing or some spectacular innovation 
to asiure commercial viability. 

The development of educational test scoring software has pro- 
gn-.ssed along the same lines. Test scoring softvv 3 has been until 
recently either an ultra expensive commercial package, an "in-house" 
project with high yearly maintenance costs or a "do-it-yourself program 
with relatively few features, minor product support and minimal growth 
potential. Many of them have been written by professionals, that is, by 
people who earn their living writing software programs. However, most 
are not of commercial quality, based on volume sales, moderate pricing, 
voluminous features, clear documentation, telephone support, ease of use. 
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design for growth, speed, application to diverse situations, and fundamen- 
tal product quality. 



The selection process for Bureau County was correctly guided by 
the needs previously identified. For the Illinois situation the central con- 
cerns have been focused on Criterion Referenced Testing (CRT) to gen- 
erate the analyses appropriate for guiding school improvement efforts. 
Other capabilities, such as item banking or graphics generation in test 
questions, are nice, but they are cosmetic features in the same category 
as word processing or contracting a print shop to make a test look more 
attractive. For Bureau County and for Illinois, the main issues arc current 
processing power to generate reports plus a solid prospect for software 
growth for future needs. 

The search for an appropriate test scoring and reporting package 
was conducted for two months by the technical staff at Educational Ser- 
vice Center No. 9. Numerous products were quickly eliminated, some 
because of high costs and substantial hardware requirements. Others 
were simply lacking in power. At least six products were reviewed in 
depth. At the end of February 1988, there were only three candidates. 

1. A software program to be written "in house". One technical 
support person was available if this became the only option. 
However, this alternative appeared less attractive as more 
information was gained on costs not only to write the program 
but also to maintain it yearly. If written "in house," the pro- 
gram would have been similar to the second candidate, but for 
a faster microcomputer. 

2. The Springfield CRT System: With a grant as an ISBE Model 
Practices Project plus substantial investment of school district 
funds, Springfield School District 186 wrote a test scoring soft- 
ware package based on several years of efforts.. The high- 
quality pedagogical efforts by Springfield School District pro- 
duced numerous reports to meet the State's mandates for 
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school improvement plans. As is clearly evident in the Spring- 
field report, Assessing Student Performance for School Im- 
provement, published in 1988 by the ISBE, that program was 
designed to meet the needs of Illinois school districts. How- 
ever, the hardware and software used by Springfield were lim- 
ited to one microcomputer that is not fast (Apple He), one 
scanner that has limited capabilities (ScanTron 1200), and one 
format of answer sheet (No. 2200). Nevertheless, with its 
exceptionally low price of less than $300, it was a software 
product worthy of consideration. 

3. The MDT Education Test Scoring System: Tliis third candi- 
date ($1095 in November 1988) was still m its pre-release ver- 
sion when the technical personnel at ESC No. 9 reviewed the 
software. Although some loose ends were evident then, the 
BCLAC chose the MDT software for its ability to satisfy the 
school district needs. Based upon the favorable experiences 
during the Model Practices Project, the BCLAC considers the 
MDT software to be quite appropriate for its current and 
anticipated needs. There is only one supplier [MDT 
Corporation, 107 Broadway, Normal, IL 61761, Telephone 
(309) 452-6388]. 

2. The Selected Software 

The MDT Educational Test Scoring System met nearly every re- 
quirement of the BCLAC. Furthermore, through a subcontract, additional 
features specifically desired for the Bureau County Model Practices Pro- 
ject were incorporated into the commercial package. 

TTie MDT Educational System originated as a method of machine 
scoring of fill-in-the-blank questions. That method uses lists of alphabet- 
ized alternative responses sufficiently long to eliminate guessing. Several 
additional advantages are discussed in a book by Anderson (1987). The 
software also handles multiple choice responses and was primarily 
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designed for general classroom testing, such as for quizzes, mid-terms and 
final examinations. Neither fill-in-the-blank questions nor general class- 
room test scoring were of immediate interest to the Bureau County Model 
Practices Project, although both aspects could become useful as the 
school improvement efforts continue. 

What Bureau County wanted (and did obtain along with the above- 
mentioned capabilities) was a powerful system to score, analyze and pro- 
duce reports from the multiple choice Criterion Referenced Test (CRT) 
described in Chapter 2. Multi-Digit Technologies Corporation, with its 
core programs already written, began work on its CRT reports in early 
1988. The initial work was influenced by the CRT model practices pro- 
ject of the Springfield Public School District 186. MDT Corporation openly 
discussed its interest in Criterion Referenced Testing wiih Dr. Robert Hill, 
Director of the Department of Instruction and the Coordinator of the 
Springfield Model Practices Project. Dr. Hill offered encouragement and 
explicitly stated that the Springfield effort could serve as a model for the 
various reports. 

The core MDT software needed only three additional reports to 
equal the Springfield capabilities. Work on those reports had already 
begun when first contact was made with the Bureau County Learning 
Assessment Cooperative. Subsequently, additional features were 
specified by the Bureau County personnel. Without question, the adoption 
of the MDT software by the Bureau County Model Practices Project 
significantly helped the Cooperative meet and maintain the established 
timetable. Likewise, the adoption greatly accelerated the incorporation of 
the CRT capabilities into the MDT software. 

Essentially, the BCLAC obtained the Illinois-related pedagogical 
quality (needs fulfillment) of the Springfield CRT methods, several addi- 
tional features, and greater speed and versatility without the costs, time 
delays, and headaches of writing an "in-house" program. Furthermore, 
the previously-discussed qualities of a commercially viable product are 
built into the programs by James v*^.. Srhoner, a systems designer of 
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exceptional talents. The quality of the MDT software is indirectly evident 
in the reports generated (see Chapter 5) and in its user friendliness. 

The MDT software is "menu driven," which means that the user 
selects the desired action from a menu-like listing (See Chapter 4). Any 
teacher or secretary who knows how to use a word processor is able to 
operate the MDT software. An assistant secretary in a Bureau County 
school district was assigned the responsibility to operate the hardware 
and software and to process 1,600 answer sheets from 50 classrooms in 
three grade levels. Her comments, printed below, are typical of what 
other users will experience: 

"The MDT Educational System leads ycu step-by-step 
through all operations performed. For those familiar with a 
computer. Lie only need for reading the manual is for quick 
reference, or to clarify an operation with which you are not 
familiar. There is no need to '''^ar making mistakes and 
therefore "ruining your data." Bac ;ig out or re-performing 
an operation is virtually as effortless as the first attempt. 
MDT software is one of the most interactive softwares 
offered and sets the user at ease as it guides you through the 
menus with helpful comments and questions. MDT software 
generates various results and reports in a variety of formats 
all of which are easily comprehended. From my point of view 
as a user/operator, this software is highly recommended!" 

"MDT Educational System software gives new meaning to 
the phrase 'user friendly.'" 

Kay Luan P. Pierson 
August 17, 1988 

More evidence of the ease of use is in Chapter 4 in which the 
actual operation of the software Is presented. 

In summary, the BCLAC is highly satisfied with the selected soft- 
ware for test scoring and reporting. 
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i. Computers 

After the needs are identified and the software is found to satisfy 
those needs, then the selection of hardware takes place. The first consid- 
eration is what computer best nms the selected software. As of the Fall 
of 1988, for reasons of speed, price, connectivity with scanners, process- 
ing of seveial reports and additional uses for the hardware, the preferred 
type of computer for the MDT software is the MS-DOS (IBM PC com- 
patible) variety. Furthermore, program enhancements and major add-on 
features will appear first on the MS-DOo version of the MDT Educational 
Test Scoring System. If an MS-DOS microcomputer is already owned 
and available or if the school district is able to purchase one (minimum 
cost of $800), then the decision on what computer to use is essentially 
finalized. Use an MS-DOS (IBM PC compatible) microcomputer for your 
CRT testing with the MDT software 




School districts could be in one of four situations: 



a. A district might own only Apple II computers and be unwill- 
ing to acquire an MS-DOS machine. Although an Apple II version 
of the core software is operational, it was designed primarily for 
typical classroom tests. The Apple version can currently (January 
1989) use only one type of scanner and it does not have all of the 
features that are described in this report for Criterion Referenced 
Testing. CRT is much more demanding of processing speed and 
larger total class sizes. Development is continuing on an Apple IIGS 
version, but a delivery date has not been set. The softv/are devel- 
oper can provide further information and configuration specifications. 

b. A district might own or lease extensive mainframe or mini- 
computer hardware. There is an available custom-made mainframe 
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version of the MDT software (in FORTRAN 77) with reduced fea- 
tures, but the MS-DOS scoring solution seems preferable. The 
MDT software permits data files in ASCII formats to be transferred 
for printing and database entries. Contact the software developer to 
discuss options. 

c. If a district owns none of the above and cannot afford to ac- 
quire an MS-DOS microcomputer, consider the cooperative option 
as discussed in Section 3.D.2. 

d. The best situation is to own or be willing tc acquire (or share) 
an MS-DOS microcomputer. 

MS-DOS (IBM PC compatible) microcomputers are manufactured 
by IBM, Tandy, Compaq, Dell, Cordata, Epson and dozens of other high 
quality suppliers of compatible PC, XT, AT, 386, and PS/2 models. 

Any currently owned MS-DOS microcomputer may be used if it 
has the minimum configuration lis^^.d below. Many school districts 
already have such computers for accounting or word processing, but 
assessment coordinators are often unaware of their availability. 

MS-DOS (IBM PC) compatibility (DOS 3.0 or higher) 

• 2 disk drives (one may be a hard disk) 

• 1 serial port (for scanner) 

• 1 parallel port (for printer) 
512K RAM memory 

• monochrome monitor 

• keyboard 

If purchase is necessary, a complete standard MS-DOS microcom- 
puter with the above listed minimum configuration can be obtained either 
from a reputable mail-order vendor for about $800.00 or from local area 
vendors who offer diverse combinations plus home-town support. Con- 
sult a technical specialist or the software vendor. Package pricing could 
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be beneficial. If sufficient funds are available, the following additional 
features for the microcomputer should be considered. The list is in 
descending order of importance. 

• Hard disk drive 
640K RAM 

AT (80286 chip) motherboard 
Graphics card (if not already included) 

Technical note: Limitations of disk space and available 
memory could impose restrictions on processing examinations 
when more than 500 students answer the same test. The 
software supplier can explain the options for handling larger 
numbers of students. 

2. Printers: 

The MDT software produces standard ASCII code that should 
print on any standard parallel printer that operates with the particular 
microcomputer. Costs range from under $200.00 to well over $1,000.00, 
depending mainly on speed and print quality. 

3. Scanners: 

The list of Optical Mark Readers (OMRs or scanners) compatible 
with the MDT software for MS-DOS microcomputers includes nearly 
every brand sold in the USA (in alphabetical order) : 

SHEET READERS • Cognitronics 

HEI 
NCS 

• ScanTron 

• Sekonic 
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CARD READERS • Chatsworth 

CR-510 
HEI 

• Michael & Assoc. 
PDI 

• True Data 

On the Apple II series microcomputers, the first scanner available 
is the CR-510 card reader. 

Most manufacturers have several models; about twenty different 
options for scanners are available. The best advice is to determine the 
needs first and then evaluate the hardware options. Packages of software 
plus hardware are often available at reduced prices. The following notes 
provide suggestions. The software supplier can provide additional infor- 
mation. 

Notes on scanners: 

a. Fundamental differences between the various scanners 
may include: 

• cost (price alone is an inadequate indicator of quality) 

• speed (through-put of answer forms per minute) 

• automatic vs manual feed (depends upon district needs) 

• connectivity to microcomputers (must be RS-232 serial) 

• darkest-mark-discriminadon (for greater accuracy in 
distinguishing light marks and erasures) (available only 
on quality sheet readers with prices above $3,000) 

• cost of answer forms (cards are less expensive but hold 
fewer responses than do sheets) 

• capacity of various types of responses per answer form 
(sheets can accommodate more responses) 

• warranty (to be expected as well as satisfaction guar- 
antee) 
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• maintenance (options usually available, including 
replacement units during repairs) 

• physical size (from small to gigantic) 

b. The HEI Model 360 sheet reader with automatic feed (price 
$3750.00) is the current (January 1989) best purchase on sheet 
readers and was used by the Bureau County Project. 

c. The CR-510 manual feed card reader (price $795.00 or less if 
packaged with the MDT software) was by far the least expensive 
yet reliable card reader. 

d. Almost any currently owned Optical Mark Reader (scanner) 
that communicates via a serial port to a microcomputer can be 
used. [Zero additional purchase cost since it is already owned. 
Contact the software vendor for compatibility information.] 

e. Card or sheet readers from NCS, ScanTron, HEI, Sekonic, 
Chatsworth, True-Data, PDI and Michael & Associates range in 
price from about $900.00 to well over $10,000.00, depending on 
capabilities. New models, features, and prices are occasionally 
released. [Contact the software vendor for compatibility 
information for specific models.] 

f. in Section D of this chapter me above hardware information 
is summarized in ways to assist school districts in their purchasing 
decisions. 
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Software, Hardware 
• and Materials 




1. Price Guidelines 

The following list is a general guideline for what might be purchased 
in order to have basic equivalence with the Bureau County Model Prac- 
tices Project as of January 1989. The total price would be substantially 
reduced if some components are already owned. 



Minimal 
ConfipMration 



More than adequate 
Configuration 



Scanner: 

CR-510 card reader 
HEI 360 sheet reader 



795 



3750 



Microcomputer: 

Minimal MS-DOS unit 
With Hard Drive and 640K 



800 



1700 



Printer: 

9 pin dot matrix 120 cps 
Faster and near letter quality 



200 



500 



Hardware Subtotal: 



$1795 



$5950 



Software 

MDT Educational System 
• Core plus CRT reports $1095 
Grade Book (for student 95 




name file with card reader) 



TOTAL 



$2985 



$7045 
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All software and hardware products used by the Bureau County 
Model Practices Project are available for purchase from commercial 
sources. Educational discounts, volume discounts, and "package" dis- 
counts often apply. Optional assistance with installation, supervision, 
training, and customized enhancements usually can be arranged to suit 
specific circumstances. 

Note: The ability to process large numbers of students means 
that the prices cited above represent a very low invest- 
ment per student served. Options for cooperative efforts 
can provide substantial savings for small school districts. 
Furthermore, all of the components can be used for addi- 
tional purposes throughout the year, thereby further 
enhancing their value to the school district. 



2. Options for Cooperative Test Scoring and Reporting 

Test scoring services have been available nationwide for many 
years. They tend to be expensive, focus on nationally administered norm- 
referenced tests and have slow turn-around time. Those services gener- 
ally lack the flexibility to handle locally designed and administered tests. 

The model of those test scoring services could be used by any 
group of school districts or cooperative entity with the necessary software 
and scanner. Some coordination and cost sharing would be necessary, 
but essentially the costs of hardware and software could be manageable 
for every school district, regardless of size. 

Note: Remember that the scanner and software have usage 
year-round for regular test scoring, attendance, and 
numerous data entry tasks. 
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To implement cooperative test scoring and reporting, several con- 
siderations are necessary: 



1. Reach agreement on the location, costs, hours of accessibility, 
and type of scanner, software and computer. 

Note: Multiple types of scanners and software could be 
offered at one location, but one is sufficient. 

Each user school district (or cooperative) needs at least one 
user/operator who is familiar with the software and system. 
Altematively, the host district could charge for operator ser- 
vices. 

2. The assessment coordinator of each district selects the ap- 
propriate answer form(s) that are suitable for the assessment 
and are compatible with the scanner and software. A variety 
of acceptable formats would be available at each scoring 
location. 

3. The user school district should be allocated an appropriate 
time slot for scanning. Based on the Bureau County experi- 
ence, scanning requires only two or three minutes per class 
of twenty-five students, using the described hardware and 
MDT software. Any user/operator who is familiar with the 
test scoring procedures discussed in Chapter 4 could scan 
the tests for five hundred students divided into twenty class- 
rooms in one to two hours. Therefore, as little as one hour 
would be needed by a very small district, or an aftemoon for 
a large (not huge) district. 

Note: This is for SCANNING ONLY. Scoring also takes 
very little time, but it can be done separately. 
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4. The joining together of classes to get building and district 
reports, plus the relative slowness of printers to make the 
reports, is a significant complication. If there is high demand 
for time with the scanner installation, a second or third copy 
of the software (but not the scanner) may be lieeucu al ihc 
cooperative scoring location. 

Alternatively, the relatively low one-time cost for the soft- 
ware might permit user school districts to have their own 
licensed copies. The software used by the Bureau County 
Model Practices Project would support an arrangement 
where scoring and report generating were done miles away 
from the scanning process. 

A cooperative effort for test scanning, scoring and reporting would 
be a logical development. Within Bureau County alone, the hardware and 
software expenditures were only a small fraction of what the twenty-two 
individual school districts would have invested for separate facilities. 
Those initially acquired items are now ava lable for use in subsequent 
years. Software enhancements and additional hardware installations can 
be added gradually in Bureau County as the need for test scoring and 
reporting increases, as certainly will occur when CRT testing for other 
learning areas is established. 

3. Recurring Expenses for Test Scoring and Reports 

Apart from the internal expenses of electricity, space, and the 
user/operator, additional expenses associated with using the described 
configuration of software and hardware are mainly for paper goods. 
Numerous formats of answer forms (cards or sheets) acceptable for 
scoring are available. Two examples are shown on the next pages. They 
vary in cost because of size and quantity. 

In general, cards cost $40.00 per thousand and sheets are $50.00 
per thousand in small quantities. Volume purchases can lower the cost by 



^ Page 62 
ERIC 



63 



3.D Cost Summary 



fifty percent (50%). Tractor-feed continuous form paper to print the 
reports is under $25.00 per box of 2500 sheets. One ribbon for a printer is 
about $6.00. Floppy diskettes cost less than fifty cents each. 

In summary, the recurring expenses for answer forms, scoring and 
reports are approximately ten to twenty cents (10 - 200) per student. The 
labor cost for the user/operator would be an equal amount. Together the 
expenses for materials and labor are estimated to cost between twenty 
and forty dollars ($20 - $40) per one hundred students. 
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Figure 3-2: Example of One Answer Card 
for 100 Multiple Choice Questions 
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Test Scoring 
from the 
Software User's 
Perspective 



A 



Getting Started 



Teachers, assessment coordinators, and users/ oper- 
ators view educational measurement software from a dif- 
ferent angle than do computer coordinators and software 
writers. The following is a user's view of testing, with ref- 
erences to the components and procedures in the MDT 
Educational System. Important features and terminology 
are introduced in this overview. For complete definitions 
and detailed instructions, see the referenced sections in the 
User's Guide of the MDT System. Ail references here are 
for MS-DOS (IBM-compatible) microcomputers with two 
floppy disk drives. Users with a hard disk have slightly 
easier operations once the directories have been created by 
the computer coordinator or software installer. 
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1. First use of the computer 



For this explanation of the MDT software we assume that the user 
(you) does know a Htfle about the computer. Power on/off switches, disk 
handhng, formatted disks, keyboard characteristics, menu usage, paper in 
the printer, etc., are not covered in this discussion. Let us also assume 
that the MDT program disks (the executable copies) have been prepared 
according to the installation instructions in Appendix A of the User's 
Guide. In other words, we are ready and able to load the MDT software 
into the computer and utilize the menu options.. 

2, The Data Disks 

When using the MDT Educational System there must always be a 
program disk in Drive A and a data disk in Drive B (unless there is a hard 
drive). Each data disk starts out as a blank formatted disk. For the CRT 
testing, it is strongly suggested that there is only one grade level and 
learning area on each disk. Think of it as a paper file folder that will 
eventually contain tests, statistics, grade book, etc., for that specific 
assessment, such as 3rd Grade Reading in 1989. Would a teacher use a 
file folder of the 6th grade for the 3rd grade class? No. Likewise, the 
data disk for a specific assessment instrument must never be removed 
from Drive B unless the computer program is at its starting point, which is 
called the Main Menu. To remove the data disk at any other point is like 
placing pages into the wrong file folder or putting the folder into the wrong 
file cabinet drawer. 

3, The MDT System and Environment 

In order to make things efficient for the teacher, the MDT software 
does a lot of work behind the scenes. There really is a SYSTEM! The 
MDT Educational System includes an ENVIRONMENT that 1 jeps track 
of all the information that is provided about each usage of a test. Details 
about the environment are written in the full User's Guide. Most users 
only need to know that the MDT Environment exists and that it is 
designed to help you in many unseen ways. 
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4. Start Up the Program 



You first turn on the computer and load in any one of the MDT 
executable program disks. If the program does not ''self-boot," type the 
three letters MDT at the system prompt, which is A> on the IBM MS- 
DOS computers. After each command, remember to press <Enter>, 
which is also known as ''carriage retum" or the " key. The first 
screen should look like Figure 4-1. 



r p" 4 MDT Educational System | ■ 

^ I Ver3ion 3.0 >" 'I 



Copyright (c) 1987 

Nlj^t: 1 -Digit Technologies (MOT) Corpora^ : or 
137 Broadway / P.O. 3ox 14 
Mor.Tal, Illinois ol''61 



MDT Cor.cept ana Design 
Softv^?are Desicn . . . . 



.Paul S. Anaerson. ?n.D. 
. . . .Jares S. Schoner 



\o part, of tnis software rray c? copiea or t:ransr.'::ted any 
:orr w.Lhcjt prior written perr:ssion frcr XDT Corporation. 

Serial \jr^ber: CCCOl 

T"^s Copy For exclusive Use By: (Vo'Jr Sc^co' Na-^e) 



>> En:er User Identifier: [ 



Figure 4-1 : User ID Screen 



5. User Identifier 



The first screen asks you to identify yourself with any three letters 
or numerals called a "User Identifier." We suggest that you use your ini- 
tials unless some other user of the computer has exactly the same initials. 
Then press the <Enter> key. The program then advances to the Main 
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Menu, which is shown in Figure 4-2. Please note that in the "Controls" 
section of the Main Menu, the letter [c] indicates an option to change the 
User Identifier. 



=! M D 
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e m ^ 



[2; 
[3; 



Test Maimer 
Test ScaL-^er 
Test Scorer 
l^ist Niaker 
Grade 3oo'< 

Report Gererator / Stancarc 
Report Generator / CRT 



la, Appena I^irectory To Data Searcn Pat" 

Ibi Rer:ove Directory Fron Data Search Pat 

.c] Cnange User Identifier 

'g^ Display Directory 

[e; Ena MDT Ecucatlonai System 



>> E'^ter Seiectiop Frcrr Mer 



vser ^aentif ler: ^ 



Figure 4-2: Main Menu of the MDT Software 
6. Main Menu 

The options offered on the Main Menu are basically self-explana- 
tory. You make your selection by typing the number or leaer and then 
<Enter>. You wi'l note that the words "Not Resident" may appear to the 
right of some options. That message means that the MDT program disk 
currently in Drive A does not contain those programs. To access one of 
them, simply remove the program disk from Drive A and insert the 
desired program disk. Type the desired menu number, then <Enter>. 
The 'Not Resident" message will blink off and the selected program will 
operate. 



P^gc 70 4.A Introduction to Costs 

ERIC 



7. Backtracking and Escape 

An important part of getting started is knowing how to stop or 
backtrack to a previous step. For MS-DOS compatibles, the ESCAPE 
<ESC> key will allow you to backtrack two different ways. When you 
have typed in information of more than one character, the <ESC> key will 
clear the characters entered. New information can be entered or the 
<ESC> key can be pressed again to return the user to the preceding step 
in the program. When the <ESC> key is used for selections that only 
require one character, you will be taken back to the preceding step in the 
program. If you repeatedly press the <ESC> key, the computer will 
eventually return to the DOS prompt. 

8. The Test Event 

When students are taught, teachers usually evaluate their learning. 
This educational measurement can be called a "lest event." The 
instmctor decides when and where it will take place. Each event is given 
a distinct identifier (name) with up to eight letters or numerals without 
spa^-es or special characters. An example would be L3R89SM, which 
couid stand for Lincoln School, third grade Reading 1989 with teacher 
Smith. Once you have decided what to call the events, you will do the 
following steps with the Test Maker program to initiate ("create") a test 
event such as L3R89SM onto the data disk in Drive B. At the Main 
Menu, make selection [1] and press <Enter> to load and begin the Test 
Maker Program. The screen is shown in Figure 4-3. [Test Maker cur- 
rently (January 1989) is only an "event maker" to which additional fea- 
tures will be added.] 
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Figure 4-3: Test Maker Screen 

The names (identifiers) of all the previously-created test events, as 
found on the data disk in Drive B, are displayed on the screen. When you 
are just beginning, that list is blank. 

Select option [C] for "Create a Test Event" and press <Enter>. 
Then type in the event name, such as L3R89SM in our example, then 
press <Enter>. The screen shown in Figure 4-4 will appear. 
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(Press ESC '^o Return To Menu) 



J 



Figure 4-4: Screen to specify a list of class 

You are allowed to specify an identifier for a Question Bank (to be 
available on a future enhancement), for an MDT list (when using the 
multi-digit format of testing), or for a class. The use of a class identifier 
will permit the software to obtain student names to match the student ID 
numbers on answer cards. Type [C] and press <Enter> for class. Then 
type the Class Identifier, such as 3LIN89 (3rd grade Lincoln School 1989) 
followed by <Enter> to record your information. Then simply press the 
<ESC> key to save the test event you just created. 

At this time you can also (and should) create in the system the test 
events for each batch of student testi, to be scored. Also create the 
events that will re",ult from the joining together of different classes that 
will take the same test, such as L3JOIN89 in Figure 4-4. Rest assured 
that any of these entries can be changed, deleted, added or renamed later. 
However, it is useful to get them into the system early because it makes 
clear to the user/operator exactly what the assessment coordinator is 
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expecting. Chapter 5 is helpful for understanding the concept and impor- 
tance of joined test events. 

9. Answer Forms and Control Forms 

Because your school probably has only one scanner (sheet or card 
reader) that will give data to the MDT Educational System, your hard- 
ware does limit the selection to a few styles of answer forms. Your 
assessment coordinator made the selection when the test was being pre- 
pared. You should be told the identifier (such as HI, H2, C2, or Tl) and 
should have a few spare forms for accidents and for the control forms.. 

Every test eveiit requires three (3) control forms.. All three must be 
on the same format of answer form (sheets or cards) used by the stu- 
dents. It is normal that the assessment coordinator would prepare these 
three forms. 

Control Form 1 is the Answer Key: Mark the correct responses. 
If a question does not have an answer marked on the key, that question 
will be skipped in all analyses. 

Control Form 2 specifies the Points per Question: Each question 
can have its own point value. For multiple choice questions, the points 
can range from one to five (A=l, B=2, etc.). If a question is to have the 
same point value as the preceding one, the point value can be left blank. 

Control Form 3 designates the Subgroup of Each Question; Each 
multiple choice question can be assigned into subgroups (learning objec- 
tives) one through five on the third control form. Later, when scoring 
takes place, any question can be placed into any subgroup from zero to 
twenty-five (0 to 25) by keyboard entry. (Subgroup zero is used as a null 
subgroup.) If a question is to be in the same subgroup as the preceding 
one, that subgroup designation can be left blank on the control form. If 
you decide not to use subgroups or do not know them at the time of 
scanning, it is imperative that a blank card or sheet be used as the third 
control form. 
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Note: The three control forms must be scanned in order and 
at the beginning of each batch of student answers, as 
described in the next section. 
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The user/operator should receive a stack of marked answer forms 
for each class that tcok the test. On top of each stack the three control 
forms must be placed. We assume that the hardware items (computer, 
scanner, and printer) are connected, turned on, and operational. Please 
be familiar with any hardware-specific instructions, such as how to place 
the answer forms into the scanner, and which buttons (if any) need to be 
pushed. 

/. Scanning 

We assume that you have to do all the test processing yourself. 
Turn on the computer with the MDT program disk for "Test Scanner" in 
Drive A and your data diskette in Drive B. Use the same three initials as 
your user identifier that was used to create the test events (see step A.5 
above). At the Main Menu (see Figure 4-2), enter the Test Scanner 
program by typing 2 and pressing <Enter>. That will present the screen 
in Figure 4-5 and the first query. When asked, specify the identifier of the 
answer form used, such as C2. Then type in the name you want for the 
test event, being sure that the name is also listed as a created test event 
in the box on the monitor screen. Press <Enter>. [If the event was not 
previously created, you need to go to the Test Maker program to create 
it.l When requested by the computer, insert your answer forms, being 
sure that the three control forms go through first and in the correct order. 
If any answer form does not match the designated format, a short "beep" 
will sound and instructions will appear on the screen. Press <ESC> when 
finished scanning. The MDT environment stores everything in the correct 
places and returns you to the Main Menu. 
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Scanner : CR-51 0 

>> Form To Scan: (C ] 



Events 



Quizl 

Quiz2 

L3R89SM 

L3R89JA 

L3R89MI 

L3JOIN89 

ALLJOIN 



>> Identifier Of Test Event To Scan:- [ 



Figure 4-5 Screen for Test Scanner 

2. Scoring 

So simple. Select the Test Scorer component on the Main Menu. 
You will then see the menu shown in Figure 4-6. Select and enter the [S] 
option to score the test event. Then type in the name of the already 
scanned test event and press the <Enter> key. You will have an opportu- 
nity to change the subgroups specified for the questions, as shown in Fig- 
ure 4-7. 
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=jTest Events|= 



Options 



[ 



(sj Score Test Event 

[:] Join Data From Two Test Event 

[e] End (Return To Main Menu) 



>> Enter Selection From Menu: [ ] 



Figure 4-6: Screen for Test S'^orer 
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Note: The lour query lines appear 
Individually as requested. 



» Change Question/Subgroup Designations (Y/N> ? ( ] 

>> Use Designations or Other Previously Scored Tests Events (Y/N) 

>> Test Event From which Designations Are To Be Extracted: ( 

■» Enter Change (ESC To Quit Ard Save Changes): ( ) 
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Figure 4-7: Screen to Accept or Change Subgroup Designations. 
Maximum of 200 questions into 25 subgroups. 
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The system asks if you want to use subgroup designations from 
another previously scored test event. The first time you must say no 
(type N, then <Enter>). For subsequent test events, you can say yes and 
then identify the test event from which the subgroup designations are to 
be retrieved. WTien you type in the changes to the subgroup designations, 
the numbers 0 (zero) through 25 are valid. Subgroup 0 (zero) will elimi- 
nate any item (question) from the analyses. When the subgroup changes 
are complete, press <ESC> to initiate the scoring process. It is quite fast. 
If you don't watch closely, you may not realize that the scoring took place. 
After the scoring process is completed the program will return automati- 
cally to the Main Menu. You may score and rescore any event as many 
times as desired. This allows you to change the subgroup designations of 
any questions as needed, an extremely important feature for the assess- 
ment coordinator who is responsible for item validity and test reliability. 

The Test Scorer component also permits the joining of two test 
events into another test event (that must have been created using the 
Test Maker program). The two events must be based on the same test, 
such as when several chsses are given the same CRT test. Joining docs 
NOT destroy the original scored events unless the same event identiHcr is 
used twice, that is, when it is being overwritten. Joining may be repeated 
until a Icurrentj maximum of 500 students are in one lest event. (To 
process larger numbers, contact the software supplier.] 

5. Reports 

Examples of the reports are found in Chapter 5. To generate re- 
ports, insert either of the two Report Generator disks and select the 
desired MDT Report Generator from the Main Menu, then press <Entcr>. 
The first monitor screen for either Report Generator is a listing of test 
event identifiers. Where prompted, type the desired test event identifier 
and press <Enter>. For the CRT Reports, one screen (sec Figure 4-8) 
handles the specification of the criterion levels for the subgroups or 
learning objectives. 
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>> Ch.jnoo Criterion Levels (Y/N) ? [ j 

>> Use Criterion Ix^vels Of Other Previously Generated Test Event (Y/N) ? { 1 
>> Test Event From Which Criterion Leve:s Are To he F.xtracied: [ 
>> Enter Subgroup Number (Or Press ESC Key To Knd Ch^inges) : ( I 

\* >> Criterion Levels (Subgroup b): ( %] 
/ 

Figure 4-8: Screen to Specify Criterion Levels 

Tlie menu of selectable reports for Standard and CRT testing have 
similar appearances except for the options in the Report Menu area (see 
Figures 4-9 and 4-10). When those menus appear, type the desired iden- 
tifying number [n] for a report and press <Enter>. The selection will be 
highlighted and marked with an asterisk "*" on the screen. 

Repeat the selection step (above) for as many reports as desired. 
To cancel a selection (to "un-selcct"), type and enter the identifying 
number a second time. All selections can be changed as many times as 
desired prior to the request to genciate reports discussed below. All 
reports arc generated in the order in which they appear in the meni:, not 
in the order ot selection. 

Note that the first two standard reports and the first CRT report are 
printed individually for each :ludent. They are intended to be given to 
each .student as personalized feedback. You are asked (query on bottom 
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of the screen) if reports are wanted for all students or just for individuals 
to be selected by Identification number. 



J M I) 7 CI J c a t ; o p a 1 S y s t. o rr t 
^ t< o V c r \ Zero: a \ or ) 



Srrros .)r ^: CJ(^r>: ;cr Nu-rors (1 :rr.)rr('cl '►Responses 

. '» ' . -^c? ^^^ by S; uc^c^i Nd'^o 

r . ,rri. — a:y c^' To: o . ar<J S..r-.^-,i. Sr(^r<vs 
H * • ( A-^ y n s 



S.'irc- ^ s:; ay 



Figure 4-9: Monitor Screen for Standiird Repons 



I-igiire 4- IT: Moiiiior screen ibr CRT Reports 
The low-:, pan of the screen is identical to Figure 4-9. 

The output mode selection is set for the screen unless you change 
it. To print on paper or create a file, type [p] or [f.l. Press <Hnter>. Type 
|sl and press <Bnter> to return to the screen option. 
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It is best to send the reports to the monitor screen the first time. 
That way you can see the results without waiting for the printer to put 
them on paper. When reports are being displayed on the screen, press 
<ESC> to interrupt a report and press any other key to proceed to the 
next report. 

Once reports and the desired output are selected, type Ig] and 
press <Enter> to generate reports. No^e that no processing occurs until 
option [g] is selected and entered to generate whatever is designated on 
the menu. More information is provided in the User's Guide. Sample 
reports are presented in Chapter 5 of this report. 

4. Grade Book 

To use the Grade Book program, you should read Section 7 of the 
User's Guide. It is easy, but there are numerous options that you might 
want to utilize. The Grade Book is of particular importance to users of 
card readers and ScanTron scanners with which there are no machine- 
readable student names on the answer forms. If names and ID numbers 
are in a grade book class and if that class is specified for the specific test 
event via the Test Maker program, then the student names will appear 
wh. ever appropriate on the generated reports. In other words, a grade 
book class will function as a database file of student names and ID num- 
bers. 

The steps discussed above rt? y are as easy as they sound. The 
hard work is done by the environment of the MDT Educational System. 
Y u, the use/operator, are protected in dozens of unseen ways from mis- 
takes so that your data remain intac^ and pure. 

The ISBE requested specific "how to do it" information in this 
report. This chapter has told you far more than what Kay Pienson knew 
when she started processing the Bureau County tests. Even so, she said 
the "MDT Educational System software gives new meaning to the phrase 
'user friendly.'" 
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Assessment 
Results 
and School 
improvement 



A 



What Does All 
This Mean? 



Everything discussed earlier in this report would be 
meaningless without this final chapter. To some extent this 
chapter represents the completion of the first round of the 
cycle of school improvement shown in Figure 1-1. Use of 
the assessment results to formulate the School Improve- 
ment Plan (SIP) to send to the ISBE appears to be the 
culmination. A cynic would say "The wishes of the 
bureaucracy have been served and it is over until next 
year."' But educators say, "The assessment results can 
help us improve the teaching and learning of the children, so 
this is really the start." 

School improvement (without the word "plan") is the 
real issue that underlies the entire cycle of efforts. The 
word "improvement" implies knowing the present level and 
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then seeking to attain a higher level. The assessment results generated 
by the cycle assist in th.; improvement process. First, the student scores 
that were so carefully gathered for each of the many CRT learning 
objectives reveal the heights and depths of the current situation. That 
information is known (or can be attained from the computer) for each 
student, classroom, teacher, building, and district, all for each learning 
objective, for each broad goal, and for each learning area at each grade 
iCvel. In the first year very little comparative data exists about previous 
equivalent populations, but at least some starting hiformation has been 
generated. The heights of accomplishment should be recognized and 
credit given where it is due.. There should be much congratulating for the 
favorable accomplishments at every level. 

The second part of improvement is the seeking of higher levels for 
the weaker areas. The CRT process permits multiple subtests to be 
viewed so that intelligent conclusions can be made to determine exactly 
for which learning objectives the improven.^.aS are to be sought. Once 
the "what to improve" is determined, then the "how to improve" can be 
formulated. Any numb'^r of factors must be considered, ranging from a 
faulty assessment instrument or curriculum guide to insufficient time-on- 
task by the student. There are too many factors to enumerate here. 
More important is the recognition that qualified educators in the school 
district will be able to examine the assessment results plus other records 
and then map out the ways to improve education. They will also send the 
SIP report to the State Board of Education, but that is only a small tip of 
the iceberg of efforts for improvement. 

To accomplish this most worthy task called school improvement, 
effort is necessary. Fortunately, the efforts discussed in Chapters 1 and 2 
make the school improvement work easier. The learning objectives are 
clear; the test is appropriate; and students have marked their answers. 
At this point the computer hardware and software must meet the needs of 
the school district and its people. Those needs are for clear, coherent, 
powerful, descriptive and analytical reports. The test scoring and report- 
ing system provides the tools and printed results. The people still do the 
final interpreting and decision making. 
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The sections which follow present the selection of reports available 
to the Bureau County school districts and others using the software. The 
sections are organized according to the person's position in the 
approximate chronological order of viewing the computer-generated 
results. Several of these persons and positions of responsibility will some- 
times reside in one person, especially in small school districts. Neverthe- 
less, the functions of the separate positions are distinct, as are the specific 
needs for data. In approximate chronological order of viewing the results, 
the positions are the following: 

1. the assessment coordinator 

2. the district administrator 

3. the district school board 

4. the State Board of Education 

5. the general public/taxpayers 

6. the classroom teachers 

7. the parents and students 

The order doe^ not imply importance. [If it did, the truly important 
ones are the lowest on the list.] Nor is there any reason to postpone giv- 
ing the applicable results to one group, for example the teachers, simply 
because another is delayed in viewing the results. Note that the early re- 
cipients of the results are focused on aggregate data. Only the teachers, 
principals, parents, and students receive and use data that identifies spe- 
cific students. 

The magnitude and interrelationships of this task are represented in 
Figure 5-1. The chart shows how each of six (6) levels of educational 
structure would utilize different selections from eight (8) computer- 
generated reports from one criterioii referenced test. For e .ample, at the 
level of a single school building (or principal), five reports are appiopriate. 
Several of those reports contain numerous parts, such as th*^ Mstcrams 
(MDT Standard Report 7) which are generated for each learning 
objective as well as for the total test. 
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Figure 5-1: Chart of Appropriate Reports for Different Uses 
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As noted in the two columns entitled ''Additional Infonnation ' and 
^'Reports Compiled," the personnel at each level also should receive an 
appropriate Summary Report of aggregate data from each higher level. 
The aggregate data summaries provide valuable ''averages" to help iden- 
tify strengths and areas for school improvement efforts. 

The total number of distinct computer-generated reports (not 
counting photocopies) needed for one CRT test depends upon the 
configuration of the entities that use the test. Figure 5-1 shows an exam- 
ple for the April 1988 experience in Bureau County. Of the sixteen 
districts (Level D) with th^rd grade classes, only one district had two ele- 
mentary school buildings (Level B). In other words, fifteen districts could 
use sixty (60) of the district ^wel reports, unaltered, for their fifteen build- 
ings, plus generating a CRT Report No. 2 for each building. The district 
with two buildings would need a new set of five reports for each building. 
The grand total for new (distinct) reports at the building level is only 
twenty-five in this example.. [Note that 60 + 25 = 85 = 17 x 5]. 

Figure 5-1 with the BCLAC example illustrates several important 
aspects of CRT testing for school improvement: 

a. Many reports (167 in the BCLAC example) are generated 
for cue test for one grade level. As a second example, a single dis- 
trict with three buildings with two classes in each could generate 
thirty-three distinct reports for one test. 

b. When calculated for three grade levels (3rd, 6th, ard 8th), 
approximately five hundred (500) distinct reports could be gener- 
ated by computer for the Language Arts assessment of the 
BCLAC, or one hundred reports for the second example. 

c. Based on the estimated cost (see Section 2.D) of $30,000 to 
prepare the three assessments, the cost per report would be $60 in 
the first year in BCLAC (or $300 in the second example). In sub- 
sequent years the costs would be only $15 (or $75 m the second 
example.) 
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d. When additional learning areas (mathematics, science, etc.) 
plus the 11 th grade classes are added to the assessment activities, 
the number of distinct reports that a school district or cooperative 
might generate could become four to eight times greater in the 
coming five years. 

Perhaps this scenario with hundreds of reports seems daunting, but 
in reality the situation is highly favorable. Consider the following addi- 
tional aspects: 

e. The computer and software are specifically designed to do 
highly repetitious tasks, so the report generation will be a relatively 
easy endeavor. 

f. The investment in software and hardware to do all these 
reports is extremely economical because the system can be used in 
successive years. These tools are within reach of most schools 
individually or through cooperatives and will save hundreds of hours 
of time for school administrators and teachers. 

g. AND ESPECIALLY IMPORTANT: A large number of 
reports appropriate to each level and with meaningful results can 
greatly enhance the prospects for significant school improvements. 
Simply stated, the reports are the key to understanding where and 
how to seek school improvement at all levels from the top where 
objectives are defined down to the individual student who needs 
specific assistance. 

As 5hown in Figure 5-1, in addition to a one-page summary per 
student, dozens of different relevant and valuable sets of information c uld 
be generated to obtain the easily extracted value of one good criterion 
referenced test. These reports should be processed and put to maximun- 
use if the intended school improvement is to be accomplished. The 
computer and software are poised to produce the reports. The remainder 
of this chapter discusses the actual usage of those reports. 

5.A What Does All This Mean? 

DO 
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To help the user maintain order among these myriad reports, the 
selected MDT software generates a "header" on each printed page. 
(See Figures 5-2 through 5-6). The date of the processing is shown. The 
"Test" name matches the identifier of the test event that was processed 
to generate the report. (The "List" and "Class" designations are 
primarily for in-course classroom testing and are not needed for the 
reports except when student names must be retrieved from a grade book 
class.) 

The test names used in several examples in this publication signify 
the "joined classes of all third grades in Bureau County" (bc3join) and 
"third grade students of V jstem Community District 306" (306we3). 

Note: Several of the reports shown in this chapter have been altered to 
illustrate some of the variety and extremes of what might occur. 
All student names are changed. Therefore, these results are not 
true representations of the Bureau County assessments in 1988. 
Annotations, size reductions, minor layout changes and several 
printers of different qualities have been used for the reports 
shown in this publication. 
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The individual, committee or consulting firm that serves as the as- 
sessment coordinator has the double duty of cleaning up the data and 
certifying that the results are meaningful. 

/. Data clean-up (MDT Standard Reports Nos. 3 and 4) 

Whenever several hundred students respond to one test in different 
rooms with different supervisors, there are possibilities of errors in the 
collection and initial processing of the data. Problems such ns marking the 
answer forms or even scanning and scoring with the wrong answer key 
can occur. The examination supervisors and the user/operator need to 
perform their duties properly; the assessment coordinator must watch 
over this and be alert to anomalies that may appear at any time in the 
analysis and reporting process. One way to delect problems of data col- 
lection or processing is to review the patterns of student answers as 
recorded by the scanner and microcomputer. This should be done on a 
class by class basis prior to any joining of the classes. Either on the mon- 
itor screen or on a paper printout, MDT Standard Report No. 3 shows 
every student response (see Figure 5-2). (Student names are from MDT 
Standard Report No. 4.) 

In MDT Standard Report No. 3 the students arc on the lines and 
items (test questions) are in the columns. Under each question number is 
the answer as recorded on the answer key. Student responses arc on 
subsequent lines. A dot (.) means the student's answer matched the 
answer key. A letter (A, B, C, D, E) is the student's answer that was 
incorrect. The coordinator should Icok for dashes (-) and blanks ( ). 
The dashes signify that the scanner/computcr/softwarc system was 
unable to distinguish between two marks the student made in the same 
question response area. This could have been intentional, a bad erasure, 
an. inadequate scanner, a misaligned sheet, or simply the mark for the 

Page 90 5.B Reports for the Assessment Coordinator 



ERIC 



92 



answer to the next question being placed in the wrong area. The 
equipment and software cannot tell the difference, but the coordinator can 
make a decision based on common sense. A few of these marks 
randomly dispersed would be normal. Many dashes throughout the report 
may indicate equipment deficiencies. Many dashes for one student could 
mean improper scanning or improper marking, such as by a student with 
deficient small motor skills needed to control the placing of the marks. 
One dash followed by a blank or by many incorrect responses that would 
have been correct for the subsequent questions indicate that the student 
possibly marked in the wrong places. 

Tost; oc3 join 
-- D -- L 1 « t : • Date: May 06. 1988 

--T-- Class:* 



Report 3: Full Listing of Responses by Label 

Student Question Number 
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Figure 5-2: MDT Stanaard Repon 3: Full Listing of Responses by Label 
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The blanks mean "no response." Blanks should not occur frequently 
in a multiple choice test. Also, many blanks for the final questions indicate 
that the student(s) did not have sufficient time to complete the test. 

What does the coordinator do in such cases? He or she must make 
a decision depending on the evidence. Ignore the few; re-scan the many; 
pull out and correct and re-scan the individual; eliminate from later analy- 
ses the questions that were not attempted; or make a list of t^e special 
case students whose individual results must be treated separately. 

The main point is that this chance to review the responses is the 
last realistic opportunity to see the students as individuals before the ag- 
gregate data analyses and reports are generated and disseminated far and 
wide. Yes, the individual results are seen again, but only much later by 
the classroom teacher, possibly at a meeting with the parents and student. 
At that time it is a little late and slightly embarrassing to discover that a 
student's CRT scores were actually in the 80s and 90s percentiles, and 
not in the 40s that the computer reported. 

Moral: A little time for review of the raw data as "seen" by .he 
computer is highly justified, especially considering the efforts already 
expended and the importance of the final results. 

2. Aggregate data vs. class data 

Once the data entry is clean for each separate class, the 
user/operator needs to join together progressively all of the classes that 
responded to the same assessment instrument. At least five meaningful 
groupings of students taking the same test can be identified and saved 
separately as the joining takes place: 

a. individual classes (Level C in Figure 5-1) 

b. all classes of the same teacher (This is common at higher 
grades, not at elementary levels.) (Level T in Figure 5-1) 
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c. all classes within one building (or under the administration of 
one principal) (Level B) 

d. all classes in a school district (Level D) 

e. all classes .>a: took the same assessment (Level A). 

Each grouping has significance to the people involved. The work of 
the assessment coordinator being discussed in this section is clearly at the 
fully aggregated level which is "A." In subsequent sections, some of the 
same reports discussed below will again be useful, bui ihe report should 
be for a specific group of students by class, teacher, building or district. 

3. Item Quality (MDT Standard Report No. 8) 

The assessment scores are ihe results of a series of test questions 
called "items." Item analysis is a well established report and the assess- 
ment coordinator uses it for the fully aggregated data. (See Figure 5-3). 
For each separate item the responses given are licked along with the 
number and percentage of students who marked that response. The cor- 
rect response that was specified on the answer key is shown with pointed 
brackets (<B>). 

The MDT software also prints the difficulty level, which by tradition 
is the decimal equivalent of the percentage of students who answered 
correctly. The value of LOGO means 100% answered correctly. 

The discrimination index is the proportion answering correctly of 
the top scoring 27% of the tested students minus the proportion answering 
correctly of the lowest scoring 27% of the tested students. The discrimi- 
nation index can range from +L0 (all top students and no lowest students 
answered coTectly) to -LO (reversed proportions). The value 0.0 means 
equal proportions answered correctly. 

Note that "discrimination'' in this statistic is scholastic and is not 
based on race, gender, religion, etc. A better term to refer to those latter 
cases is "bias" which is discussed separately in Section 5.B.8. 
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M 
D 
T 



Test 
List 
Class 



bc3 jo 1 n 



Date: May 2^, 



1968 



Report 8: Item Analysis 



Q19 Responses 


Frequency 


Percent 


- 


1 


0. 276 


A 


36 


9. 9^t5 


<B> 


260 


71 .823 


C 


32 




D 


33 


9. 116 


Dl f f leu 1 ty : 0.718 


D 1 scr 1 m 1 na 1 1 on : 


Q20 Responses 


Frequency 


Pe r cen t 




1 


0- 276 






0. 552 


<A> 


315 


67. 017 


B 


9 


2. ^j86 


C 


21 


5. 801 


D 


12 


3. 315 


E 


2 


0. 552 


Dl f f Icul ty : 0. 870 


D 1 scr 1 m 1 na 1 1 on : -» 


Q21 Responses 


Frequency 


Percent 




5 


1 . 381 


A 


35 


9. 669 


B 


87 


24. 033 


C 


79 


21 . 823 


<D> 


156 


43. 094 



DJiftculty: 0.^31 Discrimination: +0.398 

Q22 Responses Frequency Percent 

1 0.276 

A 13 3.591 

B 15 ^. 

<C> 284 78.453 

D 49 13.536 

difficulty: 0.785 Discrimination: +0.459 

Figu-e 5-3: MDT Standard Repon No. 8: Item Analysis 



Item analysis, difficulty and discrimination indices reveal a great 
deal about each test question and are valuable to the test designer. Each 
item can and should be tracked each time it is used, whether in a real test 
or in a pilot assessment. In CRT testing the difficulty levels of previously 
used items for a learning objective are useful in determining the criterion 
and expectation levels during the .selection of questions for a test. 

The balance between difficulty and discrimination merits careful 
attention. For example, if difficulty is 1.0 or 0.0, then the discrimination 
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must be 0.0. Furthemiore, very high discrimination values can only occur 
when the difficuUy levels are toward the central figures ranging from 0.3 
to 0.7 because more than 279r and less than 73% of the students would 
need to miss the question. 

Item analysis and the supporting indices are major tools for the as- 
sessment coordinator to use in the final judgment of each item. Based on 
such evidence, any unsuitable question may be eliminated prior to the 
major data processing and reporting that is sent to the other users. But 
such decisions come only after further analyses by the assessment coor- 
dinator. 

4. Statistical Summary and Histogram (MOT Standard 
Reports Nos. 6 and 7) 

Educators know they must look beyond the individual questions to- 
ward the subgroupings of questions according to the learning objectives. 
For the Assessment Coordinator, the next look is at the statistics for the 
joined test events, that is, where all students who took the test are ana- 
lyzed together. The focus is on the test and its subtests, not on the stu- 
dents. The total test scores are interesting, but they are almost irrelevant 
because the test was designed as a combination of numerous subtests. 

The MDT Standard Report No. 6 provides a statistical summary 
(See Figure 5-4) while Report No. 7 presents the data visually in a his- 
togram (Figure 5-5). These results are provided for the total test and, 
most important, separately for each learning objective. 

Much of the interpretation is obvious. A Vey points to check 
are given below. 

?. "Quantiles'' are expressed as quartiles (quarters) plus specific 
quantiles of one percent, five percent and ten percent at the top and 
bottom of the distribution. 
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b. "Extmmes" are a list of the actual scores attained by the top five 
and lowest five students. Extremes have greater importance with 
low numbers of students, as in class groups. 

c. Each asterisk represents one student. Each number is an attainable 
score, starting at the highest score possible, and "sd" means stan- 
dard deviation. 

d. The distribution is probably skewed. Expect the curve to be top- 
heavy with an extended tail to the low scores. Criterion Refer- 
enced Tests are constructed differently from Norm Referenced 
Tests with bell-shaped curves. 
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58 
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10 
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Quant i 1 es 




Ex t remes 
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Figure 5-4 MDT Standard Report No. 6: Statistical Summary 
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3 Figure 5-5: MDT Standaid Repon No. 7: Histogram 
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e. The subgroup histograms and statistics reveal the combined effect 
of the specific questions that were seen separately in the item anal- 
ysis. Single questions that are poor discriminators or highly difficult 
or simply unanswered because of lack of time can have inappropri- 
ate negative impact on statistical analyses of any complete test, and 
ghastly impact on a subgroup with only a few questions. 

i bias. (See Section 5.B.8) 

g. Student attainment of the criterion level and the class attainment of 
the expectation level can be shown. Although the computer pro- 
gram currently does not mark the criterion level on the histogram, 
each "cut-off point" of 60%, 75%, (or whatever is used) can be 
quickly determined. (The criterion level times the maximum points 
possible equals the minimum acceptable score. For example, 60% 
X 61 points = 36.6 points). By marking that criterion level on the 
histogram, the number of students (one per asterisk) who attained 
the criterion can be seen and compared with the district's expecta- 
tions. (The precisely-calculated percentage of attainment is given 
in CRT Report No. 3.) Of great importance are the number of stu- 
dents who were borderline cases either above or below the mini- 
mum acceptable score. This distribution can be readily seen on the 
histogram. Of special interest are the students within one missed or 
correct question of the criterion level, thereby having major influ- 
ence on the percentage of attainment. Such information can be 
highly revealing when subtests contain very few questions. Further 
information about the number and percentage of students who met 
the criterion level is provided in the next report to be discussed. 
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5. CRT Subgroup Summary (MDT CRT Report No. 3) 

The MDT printout called CRT Report No. 3 (not to be confused 
with Standard Report No. 3) is a summary of results for each learning 
objective. An example is in Figure 5-6. Up to 25 subgroups are permitted 
in each test event. Subgroup designations can be specified and changed 
from the Test Scorer program. An optional written descriptive name for 
each subgroup is planned for 1989. 
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Number Who 
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R i cha r d son 
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Leve I 
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Met Criterion 


Formula 20 


1 


60 


000 
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80. 110 


0 


. 813 


2 


60 


000 
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270 


74. 586 


0 


. ISO 


3 


60. 
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43. 923 


0 


. 016 
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0 
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82. 044 


0 


581 


8 


60. 


000 
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49. 724 


0 


406 


9 


60. 


000 
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316 


87. 293 


0 


449 


10 
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274 


75. 691 


0 


680 



Kudcr-R i cha r dson Formula 20 For Entire Test = 0.887 
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Number Who 
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25 
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2 


60. 000 


27 
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60. 000 
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55. 556 
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60. 000 
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1 1 


40. 741 


0. 138 
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60. 000 


27 


13 


48. 148 
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6 


60. GOO 


27 


27 


100. 000 


0. 475 


7 


60. 000 


27 


27 


100. 000 


0.012 


8 


60. 000 


27 


1 1 


40. 741 


0. 360 


9 


60. 000 


27 


25 


92. 593 


N/A 


10 


60. 000 


27 


26 


96. 296 


0. 197 


Kuder-R i char dson Formula 20 For Entire Test 


= 0.717 





Figure 5-6: MDT CRT Report No. 3: CRT Subgroup Summary Report 
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The criterion level for each separate subgroup is specified via the 
CRT Report Generator program. Levels are expressed as percentages 
(that is, 75.000 means 75%) and not as decimal percentages. 

The column for "Number Tested" is the total number of students 
processed. The current limit of 500 students in any test event will be 
raised in new releases of the software. 

The "Number Who Met Criterion" is calculated for each sub- 
group/learning objective. That number is also expressed as a percentage 
in a separate column. 

The percentage figures should be scrutinized. These numbers will 
probably be sent to the ISBE and possibly even be published in local 
newspapers and/or each school's report card. The percentages must not 
be altered after the assessment coordinator certifies that the test is valid, 
reliable, and useful for school improvement. 

6. Test Reliability (MDT CRT Report No. 3) 

The CRT Report No. 3 discussed above also provides a calculation 
of test reliability. Reliability, simply stated, is the ability of a test to provide 
similar results if the test were given a second time in similar circum- 
stances. The Kuder-Richardson Formula 20 (KR-20) is a widely re- 
spected calculation to provide a conservative estimate of the split-half 
type of reliability measurement. (See R. L. ITiomdike and E. P. Hagen, 
Measurement and Evaluation in Psychology and Education (4th edition), 
Macmillan, New York, 1977, pp 82-83, from which an extract is given on 
the next page.) The expected values of the reliability index for equally 
reliable tests (or subtests) have considerable variation depending upon the 
number of questions in each test. Acceptable index values commonly 
range from the mid 0.90's for over thirty questions to only 0.55 for tests or 
subgroups with only five questions. (See Homewood-Flossmoor, 1988, pp 
36-37.) 
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The KR-20 formula for test reliability estimated from item statistics 
(From Thorridike and Hagen, 1977, pp.82-83) is in the box below. 



where p is the proportion passing the item, and 
q is the proportion failing the item, 
n is the number of items in the test, 
s\ is the standard deviation of the test, 
E means "take the sum of and covers the n items. 



"This formula, called Kuder-Richardson Formula 20 after the names of 
the originators and the numbering in their original article, provides an es- 
timate of what is called the internal consistency of a test — the degree to 
which all of the items measure a common characteristic of the person. 
When the test is homogeneous, in the sense that every item measures the 
same general factors of ability or personality as every other item, coeffi- 
cient alpha and the Kuder-Richardson estimates have essentially the same 
interpretations as the odd-even coefficient [for split halves of test reliabil- 
ity]. The Kuder-Richardson estimate likewise (1) takes no account of 
variation in the individual from time to time, and (2) is inappropriate for 
speeded tests. Within these two limitations, it provides a conservative es- 
timate of the split-half type of reliability." 

7. Population Separation 

The MDT software is incorporating in 1989 the ability to subdivide 
any set of respondents according to any characteristic that is marked for 
scanner reading. (Alternative methods of separation are also being con- 
sidered.) This capability was specifically commissioned by the Bureau 
County Learning Assessment Cooperative for purposes of separating 
Special Education test takers from the regular student population. For 
most purposes the Special Education population is reported separately. 
The assessment coordinator should seek and follow the school district's 
guidance as to which population: should be separated. If the decision is 
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to process the two (or any other) populations separately, follow the sepa- 
ration instructions that will be provided with that software option, and then 
separately print the desired reports for each population. Note that the use 
of subpopulations can dramatically increase the number of reports to be 
generated. Careful planning is encouraged. 

8, Bias 

Bias is an extremely serious issue. The assessment coordinator 
must constantly consider the possibility of bias, search for it and eliminate 
any test items of doubtful quality. One of the best detectors of bias is the 
comparison of the item analyses for the same question according to iden- 
tified subpopulations, such as male/female or minority/non-minority stu- 
dents. The population separation routine discussed above will be very 
useful, provided the characteristics of all students are marked on the an- 
swer sheets or are otherwise accessible to the computer. 

9. Adjustments of the Assessment Instrument 

Any of several reasons could require adjustments to the test. An 
error in the answer key, a biased question, a poorly worded question, z 
shift in the subgroups, the combining of closely-related learning objectives, 
and otiier unforeseen complications could mean the elimination of a ques- 
tion or some other alteration to the subgroups that are identified for the 
learning objectives. All of the efforts discussed in this Section 5.B have 
been directed only toward certifying that the test was indeed meaningful. 
If, on the other hand, the test is found to lack validity and reliability, then it 
must not be certified and there is no purpose in further reports except to 
learn to make better assessment instruments. Alternatively, the assess- 
ment coordinator can express reservations along with basic approval of 
some or all of ihe subtests. Whatever the case, when the assessment 
coordinator can declare and justify a test (or learning objective subtest) 
as having reasonable validity and reliability, then all viewers of the sub- 
sequent reports and analyses can have reasonable confidence when using 
the test results for the improvement of the school and its students. 
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If the review and certification process of the assessment instrument 
appears to be a major task, that is correct. The consolation is that many 
of these issues were ah-eady addressed during the pilot testing of the 
questions. The curriculum team (primarily local teachers) worked hard to 
provide a bank of valid and reliable questions, from which the best were 
selected. If biased questions were eliminated in the initial rounds, the 
chance of bias in the final test items is greatly reduced. Clearly the soft- 
ware operations, the resultant reports, and the analyses described above 
can and should be conducted with the student responses to the pilot 
test(s). In other words, use the software power to process the pilot 
efforts. 

Because use of the CRT approach is relatively new for most Illinois 
school districts, some learning by the test makers is also expected. Al- 
though not optimal, the admission of need to improve the assessment ma- 
terial will probably appear on more than one School Improvement Plan. 
What is important is that the improvement is started and makes significant 
progress with each yearly cycle. 
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Reports for the 
District Administrators 



The first report to the district administrator is definitely the one pre- 
pared by the assessment coordinator. That report is referred to as Sum- 
mary A" in Figure 5-1. If on the district superintendent's desk the sign 
says, "The buck stops here," then full information about the test as an 
assessment instrument is essential. Superintendents and key assi:tants, 
including principals, should read the preceding Section 5.B. 

The second report consists of sets of statistical data, histograms, 
and the CRT summary reports for the entire district and for each school. 
(If more than one district used the same test, then global statistics should 
also be available.) Tne MDT Standard Reports Nos. 6, 7, and CRT No. 3 
shown in the previous section could be used directly. The district-level 
staff and the building-level principals need the reports to describe each 
school and to understand how that school compares with the district 
averages. Here the planning for school improvement gets underway. 
Tables of data are prepared. Lists of expectations met and not met are 
written. Special circumstances, justifications, and rationalizations of what 
has happened are worked out. An outline of strengths and deficiencies is 
formulated along with notes on where and what improvements can and 
should be made. Everything is still tentative. The principals will want to 
see the breakdown of results according to each classroom. Someone will 
remember that funding is not available to do everything at once, so priori- 
ties are suggested and materials are gathered for presentation to the 
school board. There are no formats for such tabulations and reports at 
present, but at least some guidelines will eventually come forth from the 
BCLAC or other sources. 

The administrators' use of the computer-generated results does not 
end here. Not only do they present the results to the school board and 
ISBE, the administrators also eventually coordinate and disseminate the 
results to their schools for use by classroom teachers, remedial specialists. 




Page 104 



5.C Reports for the District Administrator 

103 



parents, and students. Furthermore, through the curriculum and assess- 
ment committees, the administrators oversee the utilization of the CRT 
results in the coming cycle of LAP, curriculum guide, assessment instru- 
ment ?nd the resultant improvements evident one year later. 
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D Reports for the 
• District School Board 



f Reports for 




School 

[Improvement 



The processing was done by the administrators. The summary 
reports now go to the district school board for review, discussion and 
decisions. Additional data can be introduced, such as the results of other 
tests or reports from consultants. The options should be reasonably clear. 
Some improvements can come from better focus and revisioi.-. of the 
LAP and curriculum guide. Others relate to resource allocations, class 
sizes, statfing, and many other issues. The decisions that are made will 
directly impact the SIP report that must be sent to the ISBE. In short, the 
school board is expected to utilize the prepared results of the entire 
learning assessment process to formulate and implement school improve- 
ment in the district. 
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Report to f le Illinois 
• State Board of Education 



Rsportsforl 




School 
Improvement 



The cycle of school improvement shown in Figure 1-1 includes the 
School Improvement Plan (SIP) th '.t must be filed by each school district. 
There is no magic in the SIP. Data from CRT Report No. 3 can be re- 
ported directly. The example in Figure 5-6 (on the following two pages) is 
not intended to be a model of content, but it does show the SIP can be 
Short, Insightful and Potent. 

How each school district chooses to present its School Improve- 
ment Plan is an individual district matter governed by the regulations of 
the ISBE. Clearly the results from the CRT assessments do play a major 
role because the assessments provide much of the data upon which the 
SIP can be based. Immediately evident from the computer-generated 
reports discussed in this chapter is the wealth and variety of the data. 
Also evident is that only very small segments of the results are actually 
forwarded to the ISBE. But that which remains behind is what gives cre- 
dence to the summary results aiid provMes a base for the school district 
to undertake its improvement activities. 
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Reading Assessment Results 
November 1988 

Name of School District: 



Name of School: 



Grade: 



I. ANALYSIS OF LOCAL ASSESSMENT RESULTS 

A. Provide a summary analvsis of local readiner assessment 
results . 



SubgrouD 
Number 


Criterion 
Level 


Number 
Tested 


Number Who 
Met Criterion 


Percentaee Who 
Met. Cr i r pr i nn 


1 


60 


.000 


362 


290 


So 


. no 


2 


60 


. 000 


362 


270 


74 


586 


3 


60 


. 000 


362 


159 


43 


.923 


4 


60 


000 


362 


148 


40 


884 


5 


60 


000 


362 


189 


52 


210 


6 


60 


000 


362 


306 


84. 


530 


7 


60 


000 


362 


297 


82. 


044 


8 


60 . 


000 


362 


180 


49. 


724 


9 


60 . 


000 


362 


316 


87. 


293 


10 


60 . 


000 


362 


274 


75 . 


691 



Identify areas in which student learning met or ex ceeded 
the expected levels of performance as specified in the 
district's Learning Assessment Plan. 



No. 1 Locate information explicitly stated in the test. 

No. 2 Summarize the imoortant ideas of the text and the 
important supDorting details. 

ETC. 

Identify a-eas in which student learninEC did not meet 
exoected levels of performance as SDecified in the 
district's LearninsT Assessment Plan. 

No. 3 Ask questions and make predictions about a passage 
Drior to readinEC. 

ETC . 
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II. IMPROVEMENT ACTIVITIES 

A. List needed revisions in the district's iearnins 

obiec- ves . assessment svstem. or expectations basea on 
the anaivsis orovided in Part: I-A. 

1. Results of the BCLAC Readins test for graae 3 wiii be 
reviewed carefuilv and revised as needed. 

2. Exoectations wiii remain the same as for 1988-89. 

B. List school imorovement activities with timelines related 
to each area in which student learning did not meet 
exoectations as identified in Part I-C. 

1. A review of classroom instructional obiectives will 
be conducted to insure that aporopriate instruction 
occurs for all regular classroom students 
(9/88-5/89 ) . 

2. Remediation proecrams in reading during the regular 
school term will be made available to students 
(9/88-5/89) . 

3. Summer school programs will be provided in the area 
of remedial reading/ language arts as needed ^6/89). 

4. Homework assignments will be devised to meet 
individual student needs (9/88-5/89). 

C. If appropriate, list improvement activities with timelines 
whicS can best be addressed or implemented at the district 
level related to areas in which student learning did not 
meet expectations as identified in Part I-C. 

1. The BCLAC Reading Test for grade three will be 
analyzed and revised to reflect improvements 
indicated from the 1988 pilot year. 

2. A comprehensive K-8 language arts curriculum guide 
will be implemented beginning with the 1988-89 school 
year . 

III. REPORTING 

Indicate whether the district used the same or different 
reporting procedures from those described in the Learning 
Assessment Plan. If the district has changed its renorting 
procedures, describe the changes. 

The reporting procedures will be implemented as described in 
the August. 1987 Language Arts Learning Assessment Plan. 



5.E Report to the ISBE 



Page 109 




Reports to the General 
• Public/Taxpayers 



r 



Reports for 




School 
Improvement 



The aggregate data reports discussed thus far provide appropriate 
information to include in the school report card and to release in other 
ways to the public and taxpayers. Because the CRT assessment is es- 
sentially a local product, comparisons with other school districts are not 
possible nor even desirable. In the initial year of assessment of each ad- 
ditional learning area, the only comparisons are between the expectations 
and the actual results. However, real major value comes in the second 
and subsequent years when improvements can be demonstrated. 
Whether the school district is A+ or D- or anywhere in between, the pub- 
lic/taxpayers will be desiring demonstrable, documented improvement. 
Being local in origin and design, the school improvement activities, the 
district-wide goals, LAP, curriculum guide, and assessment instrument 
must certainly relate to the community. Therefore, the improvements 
should also relate to and be understood by the community. 
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First, the classroom teachers need appropriate copies or extracts of 
the aggregate data reports discussed above. On Figure 5-1, these would 
be Summaries A, D, and B. Teachers need to know the statistics and 
histogram distributions for the district (and maybe for the school) with 
which they can compare their own results. 

Class-level statistics and histograms are especially useful. The 
numbers of students are- much smaller, but each asterisk on the histogram 
starts to represent an identifiable student. 

CRT Report No. 3 is of diminishing imponance as the aggregate 
level reaches smaller groups. More appropriate data is available with 
greater detail in CRT Report No. 2. (See Figure 5-8, the CRT Class 
Report for Each Subgroup.) 

The purpose of CRT Report No. 2 is to infor ^hj teacher about the 
performance of each class member for each subgroup or learning objec- 
tive. Each student is named, as retrieved from the answer sheet. When 
no name is marked on the answer form, names can also be retrieved from 
the MDT gradebook class listing referenced in the Test Event. The iden- 
tification (ID) number is obtained from the answer sheet or card.. Scores 
of attained points (raw) are shown alongside the maximum points possi- 
ble. That fraction is converted into the percent attained by each student. 
The criterion level is specified at the time of processing the CRT reports 
and is shown near the top of the report. Any student who does not attain 
that criterion level is identified with an asterisk. Sorting the report by 
name, ID number or ranked score will present the results in three useful 
ways. 



ERIC 



5.G Reports for the Classroom Teacher 

113 



Page 111 



Test : 306we3 
List : « 
Class: Uest3 



Date: May 12, 1988 



Report 2: CRT Class Report For Each Subgroup 

Subgroup 2 
Criterion Level = 60.000 

Name ID Score Percent « = Below 

Number Raw/Max Attained Criterion Level 



ALBERT CHRISTY B 


306 


101 


6/6 


100. 000 




ALEXANDER ROBERT V 


306 


201 


5/6 


83. 333 




ALTES CHARLES R 


306 


202 


4/6 


66.667 




BATES DONALD M 


306 


104 


5/6 


83. 333 




COOPER MICHAEL J 


306 


203 


6/6 


100.000 




DETWEILER BRIAN L 


306 


204 


2/6 


33.333 




DUNAWAY DAVID S 


306 


105 


6/6 


100.000 




EATON GREGORY M 


306 


106 


3/6 


50.000 




ERUIN DALE E 


306 


107 


6/6 


100.000 




FERNALD RONALD K 


306 


206 


5/6 


83. 333 




GEISELMAN DEBRA S 


306 


207 


6/6 


100.000 




HAGAN JOHN K 


306 


108 


3/6 


50. 000 





I Choice Of Sorted Order ^» 
Cn] Name 
C i 1 ID Number 
Cs] Score (Subgroup) 
>> Enter Selection From Menu: C 3 



Figure 5-8: MDT CRT Report No. 2: CRT Class Report for Each Subgroup. 
These assessment results have several purposes for the teachers: 

a. The teachers can identify specific learning objectives that are 
well covered and others that merit greater attention to raise 
scores of the entire class next year. 

b. The teachers can identify specific students who need additional 
assistance on specific learning objectives. If results are avail- 
able during the current enrollment session, the teacher should 
use them to aid the student. In many cases the additional assis- 
tance will be provided by other teachers who will instruct the 
student in the next academic year. Those teachers should be 
provided assessment data about their incoming classes. 
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Note that the actual test questions were not issued to the teacher 
(nor to the parents or students). In addition to maintaining test security, 
confidential control of the assessment items helps diminish the temptation 
of "teaching to the test." The CRT assessment deals with clearly-stated 
learning objectives that are more important for school improvement than 
are individual questions. For these reasons there is little value in , -oviding 
the Item Analysis report (Standard Report No. 8) for the classroom 
teacher. However, the building principal might use the item analysis with 
the teacher to identify strengths and deficiencies on specific topics. 
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H Reports to the 
• Parents and Students 



The parents and students are listed together, but the emphasis is on 
the students. CRT assessment is not designed to provide meaningful 
score differentiations among 'he top half of the students. Those students 
exceeded the criterion level, period. Therefore, for many students and 
their parents the message from the CRT assessment is simply that the 
criterion levels were met. 



Reports for 




School 
ImDrovement 



For students who either did not attain the criterion level or were 
marginal, the CRT results can provide a tremendous diagnostic value. 
Since each test was essentially many subtests with specified learning ob- 
jectives, the feedback to the student, parent and teacher is quite detailed. 
CRT Report No. 1 of the MDT System provides a separate page for each 
student (See Figure 5-9). 





T e s 


t 


: 506 we3 








D 


L i s 


t 








Date: May 24, 1?88 


T 


C 1 a 


s 


s: Uest3 














Repor t 1 : 


CRT Student 


Repor t 


Name: DOE 


JOHN C 








Total Score: 44 / 61 (Maximum) 


ID Number 


306 106 












Subg r cup 


Subtotal Score 




Percent 


Criterion 


* = Below 


ID 


Raw/Max 




Attained 


Level 


Criterion Level 


1 


7/7 




100.000 


60 


000 




2 


3/6 




50.000 


60 


000 




3 


A/5 




80. 000 


60 


000 




A 


2/A 




50.000 


60 


000 




5 


3/6 




50.000 


60. 


000 




6 


7/10 




70.000 


60, 


000 




7 


6/6 




100.000 


60, 


000 




8 


3/7 




42. 857 


60, 


000 




9 


A/A 




100.000 


60. 


000 




10 


5/6 




83.333 


60. 


000 





Figure 5-9: MDT CRT Report No. 1 : CRT Student Report 



o Page 114 

ERIC 



5.H. Reports to the Parents and Students 

115 



Each subgroup has one line and (in the future) each can have a 
printed learning objective name. The subtotal score and the maximum 
points possible are shown for each subgroup. This is converted into per- 
cent attained. The criterion level is shown, along with an asterisk if the 
student's score was below the criterion level for the subgroup. Utilizing 
this information, the teacher and possibly others on the school staff can 
formulate a plan for remedial study. 

Note: The MDT Educational Test Scoring System generates additional 
reports that are of major value for in-course testing but only 
would be used in the CRT situation if copies of the test were to 
be openly distributed to students, parents and the public. Stan- 
dard Reports Nos. 1 and 2 identify each question the student 
missed. Standard Reports Nos. 4 and 5 list the students by name 
and/or ID number, plus their scores. Explanations and examples 
of these reports are printed in the Us :r's Guide of the MDT 
System. 
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Conclusion 



The cycle has gone full-circle when the results of 
one year's assessments have impact upon the subsequent 
round of planning, instruction, assessment and reporting. 
Tlie process is impressive in its simplicity. 

Throughout Illinois and the nation much work 
remains to be done. Many years ar? needed to assess the 
impact of "goals-directed" education. Fast answers are 
not possible, and deviation now from the chosen path is 
not advisable. The initial efforts indicate desirable results. 
In Illinois, the process has bes;un in full force. 

Concerning the Bureau County experiences, the 
outcomes of the project are not only positive, they far 
exceed the initial expectations. As a result of hard work 
and fortuitous combinations of State support, regional 
leadership, committee members, consultants and suppliers. 
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an assessment methodology appropriate to Illinois' needs has evolved. By 
all measures used, it appears that this project can indeed serve as a 
model. 

The two major divisions of the Bureau County Model Practices 
Project are closely related but quite separate: 

(a) the construction of the assessment instrument, and 

(b) the scoring and reporting mechanism of software and hard- 
ware. 

The test construction and test scoring described in this report are 
NOT dependent upon each other. In other words, the assessment instru- 
ments and curriculum guide from Bureau County could be used with other 
software packages. Likewise, the selected MDT software package can 
process assessment results from any assessment instrument for which the 
student responses are marked on an appropriate answer card or sheet. 
However, coordination of the two major divisions is very important, as is 
illustrated in the Bureau County experience. Planning is essential. 

Within Bureau County the impact is already noteworthy. The April 
1988 results are actually usable, not merely a "dry run" learning experi- 
ence. Enthusiasm is mounting for the 1989 language arts assessment. 
The decision has recently been made by the school districts in the coop- 
erative to extend the assessment and reporting process to all grade levels, 
not just the mandated 3, 6, 8, and 11 levels. The steering committee and 
consultants are currently exploring the possibility of expanding the report- 
ing process to include a diagnostic/prescriptive component in the software 
package. The power of computer-assisted analyses is just beginning to 
be tapped. 

It is important to note that assessment other than by machine- 
scored tests may be used. The Bureau County Cooperarive and many 
other school districts are actively pursuing assessment components with 
writing samples, oral lesponses, and longitudinal observation of the stu- 
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dents by qualified teachers. The Bureau County efforts with the above- 
mentioned non-test methods include an additional component: How can 
the speed, cost savings, and statistical analysis power of computers and 
optical mark readers (scanners) be incoq^orated into the non-test methods 
of assessment? At present it is sufficient to state that the Bureau County 
Learning Assessment Cooperative feels strongly that computer-related 
technology has a diverse and important role in the entire arena of educa- 
tional measurement. 

Also, the word of success is spreading among the committees 
responsible for mathematics, biological and physical sciences, and social 
sciences. The ability to see the formats of the data and reports is a real 
advantage for those responsible for subsequent assessments. 

For the Bureau County Learning Assessment Cooperative 
(BCLAC), this report is the conclusion of the initial "pilot'* phases. The 
BCLAC is proceeding to the next stages. Additional computer-generated 
analyses and reports plus item banks for the diverse learning areas are 
being developed. Any school district desiring information or considering 
adoption of a similar route are cordially invit^^d to contact Mr. Larry 
Marsh to explore avenues of mutual assistance. All of us share the 
common goal of excellence in education. 
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